{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 金融风控-贷款违约预测 特征工程\n",
    "\n",
    "### 学习任务\n",
    "\n",
    "学习特征预处理、缺失值、异常值处理、数据分桶等特征处理方法\n",
    "\n",
    "学习特征交互、编码、选择的相应方法\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数学基础知识\n",
    "\n",
    "#### 1.剔除异常值的方法（参考：https://zhuanlan.zhihu.com/p/65023844）\n",
    "\n",
    "**正态分布**\n",
    "\n",
    "如果数据近似服从正态分布，那么其数据关于平均值μ对称，数值分布在（μ-σ，μ+σ）中的概率为0.6827，数值分布在（μ-3σ，μ+3σ）中的概率为0.9973。也就是说落在3σ之外是小概率事件，依据这个原理，把数据集中超过3σ的数据都删去。\n",
    "\n",
    "剔除的核心语句是\n",
    "```\n",
    "outliers_cut_off = data_std * 3\n",
    "```\n",
    "**Tukey箱型图法**\n",
    "\n",
    "由于极端值还会影响正态分布的均值和标准差，所以用分位数来代替均方差。\n",
    "\n",
    "如图所示，四分位间距Q3-Q1，(Q1, Q3)涵盖了数据分布最中间的50%的数据，具有稳健性。数据落在 (Q1-1.5*IQR, Q3+1.5*IQR) 范围内，则认为是正常值，在此范围之外的即为异常值。\n",
    "\n",
    "![avatar](https://pic3.zhimg.com/80/v2-1cd5463f6346a73af2573dfaecfbe0bb_1440w.jpg)\n",
    "\n",
    "**不是标准正态分布**\n",
    "\n",
    "当原始数据的分布是有偏的，不满足正态分布时，可通过BOX-COX转化，在一定程度上修正分布的偏态。需要搜寻最优参数λ：\n",
    "\n",
    "![avatar](https://pic4.zhimg.com/80/v2-dd0e95e24f06819033953a6e2d88fded_1440w.jpg)\n",
    "\n",
    "以下是一个转换的例子：\n",
    "\n",
    "![avatar](https://pic4.zhimg.com/80/v2-51947f1d5b5f7fb194f14b328828a497_1440w.jpg)\n",
    "\n",
    "\n",
    "#### 2.协方差\n",
    "\n",
    "衡量两个变量的总体误差，也就是两个变量的变化趋势之间的关系\n",
    "\n",
    "Cov(X,Y) = E[(X-E(X))(Y-E(Y))] = E(XY) - E(X)E(Y)\n",
    "\n",
    "相关性ρ(XY) = Cov(X,Y)/(D(X)D(Y)^0.5\n",
    "\n",
    "corrwith()是pandas中用于计算两个dataframe行列之间的相关性的函数\n",
    "\n",
    "输入可以是DataFrame, Series；\n",
    "\n",
    "参数axis用于指示是行还是列\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### api用法\n",
    "\n",
    "#### 1.填充数据fillna()\n",
    "\n",
    "参考：https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html?highlight=fillna#pandas.DataFrame.fillna\n",
    "\n",
    "常值填充：数值、字典、Pandas的Series或者DataFrame\n",
    "\n",
    "注意inplace=true时直接替换原值\n",
    "```\n",
    "data.fillna(0)\n",
    "data.fillna(0,inplace=True)\n",
    "#填充一个字典\n",
    "data.fillna({0:10,1:20,2:30})\n",
    "```\n",
    "使用附近值填充，method{‘backfill’, ‘bfill’, ‘pad’, ‘ffill’}\n",
    "```\n",
    "data.fillna(method='ffill')\n",
    "```\n",
    "限制填充个数：limit=int,也就是指定往某个方向填充几个数（默认填满）\n",
    "```\n",
    "data.fillna(value=values, limit=1)\n",
    "```\n",
    "axis{0 or ‘index’, 1 or ‘columns’} 指定是横着填还是竖着填\n",
    "```\n",
    "data.fillna(value=values,axis=1)\n",
    "```\n",
    "#### 2.将各类标签转换为数字\n",
    "\n",
    ".map{dic}:通过map函数把每一个标签转换成对应的数字，在个人信用评级中将ABCD转换为1234\n",
    "```\n",
    "for data in [data_train, data_test_a]:\n",
    "    data['grade'] = data['grade'].map({'A':1,'B':2,'C':3,'D':4,'E':5,'F':6,'G':7})\n",
    "```\n",
    "\n",
    "pd.get_dummies():对特征进行one-hot编码，\n",
    "\n",
    "参考：https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html\n",
    "    https://www.cnblogs.com/lianyingteng/p/7792693.html\n",
    "    \n",
    "通俗的说，one-hot编码是一串很稀疏的向量，只有在某一项对应的位置不为零\n",
    "\n",
    "关键参数有\n",
    "\n",
    "data：array-like, Series, or DataFrame，传入的参数\n",
    "\n",
    "prefix：str, list of str, or dict of str, default None，前缀的名称，追加传入数据的名称\n",
    "\n",
    "返回值的数据类型是DataFrame\n",
    "\n",
    "用此方法转换了信用评级子分数、房产、评估状态、贷款目的、贷款地的参数\n",
    "```\n",
    "for data in [data_train, data_test_a]:\n",
    "    data = pd.get_dummies(data, columns=['subGrade', 'homeOwnership', 'verificationStatus', 'purpose', 'regionCode'], drop_first=True)\n",
    "```\n",
    "#### 3.sklearn中的特征选择方法\n",
    "\n",
    "首先进入官网的api参考：https://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_selection\n",
    "\n",
    "**VarianceThreshold**\n",
    "\n",
    "方差选择法中，先要计算各个特征的方差，然后根据设定的阈值，选择方差大于阈值的特征\n",
    "\n",
    "This feature selection algorithm looks only at the features (X), not the desired outputs (y), and can thus be used for unsupervised learning.\n",
    "\n",
    "简而言之就是对输入数据X进行计算,不对输出y做处理\n",
    "\n",
    "|方法|作用|\n",
    "|--|--|\n",
    "fit(X[, y]) |Learn empirical variances from X.从X学习经验方差\n",
    "fit_transform(X[, y])| Fit to data, then transform it.拟合XY\n",
    "get_params([deep])|Get parameters for this estimator.\n",
    "get_support([indices])|Get a mask, or integer index, of the features selected\n",
    "inverse_transform(X)|Reverse the transformation operation\n",
    "set_params(**params)|Set the parameters of this estimator.\n",
    "transform(X)|Reduce X to the selected features.\n",
    "\n",
    "**SelectKBest**\n",
    "\n",
    "taking two arrays X and y, and returning a pair of arrays (scores, pvalues) or a single array with scores\n",
    "\n",
    "输入X和Y，返回一组array\n",
    "\n",
    "|方法|作用|\n",
    "|--|--|\n",
    "fit(X[, y]) |Learn empirical variances from X.\n",
    "fit_transform(X[, y])| Fit to data, then transform it.\n",
    "get_params([deep])|Get parameters for this estimator.\n",
    "get_params(deep=True)[source]|Get parameters for this estimator.\n",
    "get_support(indices=False)|Get a mask, or integer index, of the features selected\n",
    "inverse_transform(X)|Reverse the transformation operation\n",
    "\n",
    "**RFE**\n",
    "\n",
    "Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.\n",
    "\n",
    "递归消除特征法使用一个基模型来进行多轮训练，每轮训练后，消除若干权值系数的特征，再基于新的特征集进行下一轮训练。\n",
    "\n",
    "通俗解释就是，递归特征消除（RFE）的目标是通过递归考虑越来越少的特征集来选择特征，每次把最没用的特征剪掉，直到得到理想模型\n",
    "\n",
    "|方法|作用|\n",
    "|--|--|\n",
    "decision_function(X)|Compute the decision function of X.\n",
    "fit(X, y)|Fit the RFE model and then the underlying estimator on the selected\n",
    "fit_transform(X[, y])|Fit to data, then transform it.\n",
    "get_params([deep])|Get parameters for this estimator.\n",
    "get_support([indices])|Get a mask, or integer index, of the features selected\n",
    "inverse_transform(X)|Reverse the transformation operation\n",
    "predict(X)|Reduce X to the selected features and then predict using the\n",
    "predict_log_proba(X)|Predict class log-probabilities for X.\n",
    "predict_proba(X)|Predict class probabilities for X.\n",
    "score(X, y)|Reduce X to the selected features and then return the score of the\n",
    "set_params(**params)|Set the parameters of this estimator.\n",
    "transform(X)|Reduce X to the selected features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.首先导入需要用到的库，并读入数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import datetime\n",
    "from tqdm import tqdm\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "from catboost import CatBoostRegressor\n",
    "import warnings\n",
    "from sklearn.model_selection import StratifiedKFold, KFold\n",
    "from sklearn.metrics import accuracy_score, f1_score, roc_auc_score, log_loss\n",
    "warnings.filterwarnings('ignore')\n",
    "data_train =pd.read_csv('train.csv')\n",
    "data_test_a = pd.read_csv('testA.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是将对缺失值的处理部分，其中用到的api有isnull(), "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                        0\n",
       "loanAmnt                  0\n",
       "term                      0\n",
       "interestRate              0\n",
       "installment               0\n",
       "grade                     0\n",
       "subGrade                  0\n",
       "employmentTitle           1\n",
       "employmentLength      46799\n",
       "homeOwnership             0\n",
       "annualIncome              0\n",
       "verificationStatus        0\n",
       "issueDate                 0\n",
       "isDefault                 0\n",
       "purpose                   0\n",
       "postCode                  1\n",
       "regionCode                0\n",
       "dti                     239\n",
       "delinquency_2years        0\n",
       "ficoRangeLow              0\n",
       "ficoRangeHigh             0\n",
       "openAcc                   0\n",
       "pubRec                    0\n",
       "pubRecBankruptcies      405\n",
       "revolBal                  0\n",
       "revolUtil               531\n",
       "totalAcc                  0\n",
       "initialListStatus         0\n",
       "applicationType           0\n",
       "earliesCreditLine         0\n",
       "title                     1\n",
       "policyCode                0\n",
       "n0                    40270\n",
       "n1                    40270\n",
       "n2                    40270\n",
       "n2.1                  40270\n",
       "n4                    33239\n",
       "n5                    40270\n",
       "n6                    40270\n",
       "n7                    40270\n",
       "n8                    40271\n",
       "n9                    40270\n",
       "n10                   33239\n",
       "n11                   69752\n",
       "n12                   40270\n",
       "n13                   40270\n",
       "n14                   40270\n",
       "dtype: int64"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "numerical_fea = list(data_train.select_dtypes(exclude=['object']).columns)\n",
    "category_fea = list(filter(lambda x: x not in numerical_fea,list(data_train.columns)))\n",
    "label = 'isDefault'\n",
    "numerical_fea.remove(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "#按照平均数填充数值型特征\n",
    "data_train[numerical_fea] = data_train[numerical_fea].fillna(data_train[numerical_fea].median())\n",
    "data_test_a[numerical_fea] = data_test_a[numerical_fea].fillna(data_train[numerical_fea].median())\n",
    "#按照众数填充类别型特征\n",
    "#data_train[category_fea] = data_train[category_fea].fillna(data_train[category_fea].mode())\n",
    "#data_test_a[category_fea] = data_test_a[category_fea].fillna(data_train[category_fea].mode())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                        0\n",
       "loanAmnt                  0\n",
       "term                      0\n",
       "interestRate              0\n",
       "installment               0\n",
       "grade                     0\n",
       "subGrade                  0\n",
       "employmentTitle           0\n",
       "employmentLength      46799\n",
       "homeOwnership             0\n",
       "annualIncome              0\n",
       "verificationStatus        0\n",
       "issueDate                 0\n",
       "isDefault                 0\n",
       "purpose                   0\n",
       "postCode                  0\n",
       "regionCode                0\n",
       "dti                       0\n",
       "delinquency_2years        0\n",
       "ficoRangeLow              0\n",
       "ficoRangeHigh             0\n",
       "openAcc                   0\n",
       "pubRec                    0\n",
       "pubRecBankruptcies        0\n",
       "revolBal                  0\n",
       "revolUtil                 0\n",
       "totalAcc                  0\n",
       "initialListStatus         0\n",
       "applicationType           0\n",
       "earliesCreditLine         0\n",
       "title                     0\n",
       "policyCode                0\n",
       "n0                        0\n",
       "n1                        0\n",
       "n2                        0\n",
       "n2.1                      0\n",
       "n4                        0\n",
       "n5                        0\n",
       "n6                        0\n",
       "n7                        0\n",
       "n8                        0\n",
       "n9                        0\n",
       "n10                       0\n",
       "n11                       0\n",
       "n12                       0\n",
       "n13                       0\n",
       "n14                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['grade', 'subGrade', 'employmentLength', 'issueDate', 'earliesCreditLine']"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_fea"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "#转化成时间格式\n",
    "for data in [data_train, data_test_a]:\n",
    "    data['issueDate'] = pd.to_datetime(data['issueDate'],format='%Y-%m-%d')\n",
    "    startdate = datetime.datetime.strptime('2007-06-01', '%Y-%m-%d')\n",
    "    #构造时间特征\n",
    "    data['issueDateDT'] = data['issueDate'].apply(lambda x: x-startdate).dt.days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1 year        52489\n",
       "10+ years    262753\n",
       "2 years       72358\n",
       "3 years       64152\n",
       "4 years       47985\n",
       "5 years       50102\n",
       "6 years       37254\n",
       "7 years       35407\n",
       "8 years       36192\n",
       "9 years       30272\n",
       "< 1 year      64237\n",
       "NaN           46799\n",
       "Name: employmentLength, dtype: int64"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['employmentLength'].value_counts(dropna=False).sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def employmentLength_to_int(s):\n",
    "    if pd.isnull(s):\n",
    "        return s\n",
    "    else:\n",
    "        return np.int8(s.split()[0])\n",
    "for data in [data_train, data_test_a]:\n",
    "    data['employmentLength'].replace(to_replace='10+ years', value='10 years', inplace=True)\n",
    "    data['employmentLength'].replace('< 1 year', '0 years', inplace=True)\n",
    "    data['employmentLength'] = data['employmentLength'].apply(employmentLength_to_int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0     15989\n",
       "1.0     13182\n",
       "2.0     18207\n",
       "3.0     16011\n",
       "4.0     11833\n",
       "5.0     12543\n",
       "6.0      9328\n",
       "7.0      8823\n",
       "8.0      8976\n",
       "9.0      7594\n",
       "10.0    65772\n",
       "NaN     11742\n",
       "Name: employmentLength, dtype: int64"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['employmentLength'].value_counts(dropna=False).sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "312537    Aug-2007\n",
       "294453    Jan-2005\n",
       "362167    Dec-1975\n",
       "566062    Jul-2004\n",
       "691766    Oct-2006\n",
       "Name: earliesCreditLine, dtype: object"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train['earliesCreditLine'].sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data in [data_train, data_test_a]:\n",
    "    data['earliesCreditLine'] = data['earliesCreditLine'].apply(lambda s: int(s[-4:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grade 类型数： 7\n",
      "subGrade 类型数： 35\n",
      "employmentTitle 类型数： 79282\n",
      "homeOwnership 类型数： 6\n",
      "verificationStatus 类型数： 3\n",
      "purpose 类型数： 14\n",
      "postCode 类型数： 889\n",
      "regionCode 类型数： 51\n",
      "applicationType 类型数： 2\n",
      "initialListStatus 类型数： 2\n",
      "title 类型数： 12058\n",
      "policyCode 类型数： 1\n"
     ]
    }
   ],
   "source": [
    "# 部分类别特征\n",
    "cate_features = ['grade', 'subGrade', 'employmentTitle', 'homeOwnership', 'verificationStatus', 'purpose', 'postCode', 'regionCode', \\\n",
    "                 'applicationType', 'initialListStatus', 'title', 'policyCode']\n",
    "for f in cate_features:\n",
    "    print(f, '类型数：', data[f].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data in [data_train, data_test_a]:\n",
    "    data['grade'] = data['grade'].map({'A':1,'B':2,'C':3,'D':4,'E':5,'F':6,'G':7})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 类型数在2之上，又不是高维稀疏的,且纯分类特征\n",
    "for data in [data_train, data_test_a]:\n",
    "    data = pd.get_dummies(data, columns=['subGrade', 'homeOwnership', 'verificationStatus', 'purpose', 'regionCode'], drop_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 异常值处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_outliers_by_3segama(data,fea):\n",
    "    data_std = np.std(data[fea])\n",
    "    data_mean = np.mean(data[fea])\n",
    "    outliers_cut_off = data_std * 3\n",
    "    lower_rule = data_mean - outliers_cut_off\n",
    "    upper_rule = data_mean + outliers_cut_off\n",
    "    data[fea+'_outliers'] = data[fea].apply(lambda x:str('异常值') if x > upper_rule or x < lower_rule else '正常值')\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正常值    800000\n",
      "Name: id_outliers, dtype: int64\n",
      "id_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: loanAmnt_outliers, dtype: int64\n",
      "loanAmnt_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: term_outliers, dtype: int64\n",
      "term_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    794259\n",
      "异常值      5741\n",
      "Name: interestRate_outliers, dtype: int64\n",
      "interestRate_outliers\n",
      "异常值      2916\n",
      "正常值    156694\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    792046\n",
      "异常值      7954\n",
      "Name: installment_outliers, dtype: int64\n",
      "installment_outliers\n",
      "异常值      2152\n",
      "正常值    157458\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: employmentTitle_outliers, dtype: int64\n",
      "employmentTitle_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    799701\n",
      "异常值       299\n",
      "Name: homeOwnership_outliers, dtype: int64\n",
      "homeOwnership_outliers\n",
      "异常值        62\n",
      "正常值    159548\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    793973\n",
      "异常值      6027\n",
      "Name: annualIncome_outliers, dtype: int64\n",
      "annualIncome_outliers\n",
      "异常值       756\n",
      "正常值    158854\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: verificationStatus_outliers, dtype: int64\n",
      "verificationStatus_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    783003\n",
      "异常值     16997\n",
      "Name: purpose_outliers, dtype: int64\n",
      "purpose_outliers\n",
      "异常值      3635\n",
      "正常值    155975\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    798931\n",
      "异常值      1069\n",
      "Name: postCode_outliers, dtype: int64\n",
      "postCode_outliers\n",
      "异常值       221\n",
      "正常值    159389\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    799994\n",
      "异常值         6\n",
      "Name: regionCode_outliers, dtype: int64\n",
      "regionCode_outliers\n",
      "异常值         1\n",
      "正常值    159609\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    798440\n",
      "异常值      1560\n",
      "Name: dti_outliers, dtype: int64\n",
      "dti_outliers\n",
      "异常值       466\n",
      "正常值    159144\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    778245\n",
      "异常值     21755\n",
      "Name: delinquency_2years_outliers, dtype: int64\n",
      "delinquency_2years_outliers\n",
      "异常值      5089\n",
      "正常值    154521\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788261\n",
      "异常值     11739\n",
      "Name: ficoRangeLow_outliers, dtype: int64\n",
      "ficoRangeLow_outliers\n",
      "异常值       778\n",
      "正常值    158832\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788261\n",
      "异常值     11739\n",
      "Name: ficoRangeHigh_outliers, dtype: int64\n",
      "ficoRangeHigh_outliers\n",
      "异常值       778\n",
      "正常值    158832\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    790889\n",
      "异常值      9111\n",
      "Name: openAcc_outliers, dtype: int64\n",
      "openAcc_outliers\n",
      "异常值      2195\n",
      "正常值    157415\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    792471\n",
      "异常值      7529\n",
      "Name: pubRec_outliers, dtype: int64\n",
      "pubRec_outliers\n",
      "异常值      1701\n",
      "正常值    157909\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    794120\n",
      "异常值      5880\n",
      "Name: pubRecBankruptcies_outliers, dtype: int64\n",
      "pubRecBankruptcies_outliers\n",
      "异常值      1423\n",
      "正常值    158187\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    790001\n",
      "异常值      9999\n",
      "Name: revolBal_outliers, dtype: int64\n",
      "revolBal_outliers\n",
      "异常值      1359\n",
      "正常值    158251\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    799948\n",
      "异常值        52\n",
      "Name: revolUtil_outliers, dtype: int64\n",
      "revolUtil_outliers\n",
      "异常值        23\n",
      "正常值    159587\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    791663\n",
      "异常值      8337\n",
      "Name: totalAcc_outliers, dtype: int64\n",
      "totalAcc_outliers\n",
      "异常值      1668\n",
      "正常值    157942\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: initialListStatus_outliers, dtype: int64\n",
      "initialListStatus_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    784586\n",
      "异常值     15414\n",
      "Name: applicationType_outliers, dtype: int64\n",
      "applicationType_outliers\n",
      "异常值      3875\n",
      "正常值    155735\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    775134\n",
      "异常值     24866\n",
      "Name: title_outliers, dtype: int64\n",
      "title_outliers\n",
      "异常值      3900\n",
      "正常值    155710\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    800000\n",
      "Name: policyCode_outliers, dtype: int64\n",
      "policyCode_outliers\n",
      "正常值    159610\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    782773\n",
      "异常值     17227\n",
      "Name: n0_outliers, dtype: int64\n",
      "n0_outliers\n",
      "异常值      3485\n",
      "正常值    156125\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    790500\n",
      "异常值      9500\n",
      "Name: n1_outliers, dtype: int64\n",
      "n1_outliers\n",
      "异常值      2491\n",
      "正常值    157119\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    789067\n",
      "异常值     10933\n",
      "Name: n2_outliers, dtype: int64\n",
      "n2_outliers\n",
      "异常值      3205\n",
      "正常值    156405\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    789067\n",
      "异常值     10933\n",
      "Name: n2.1_outliers, dtype: int64\n",
      "n2.1_outliers\n",
      "异常值      3205\n",
      "正常值    156405\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788660\n",
      "异常值     11340\n",
      "Name: n4_outliers, dtype: int64\n",
      "n4_outliers\n",
      "异常值      2476\n",
      "正常值    157134\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    790355\n",
      "异常值      9645\n",
      "Name: n5_outliers, dtype: int64\n",
      "n5_outliers\n",
      "异常值      1858\n",
      "正常值    157752\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    786006\n",
      "异常值     13994\n",
      "Name: n6_outliers, dtype: int64\n",
      "n6_outliers\n",
      "异常值      3182\n",
      "正常值    156428\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788430\n",
      "异常值     11570\n",
      "Name: n7_outliers, dtype: int64\n",
      "n7_outliers\n",
      "异常值      2746\n",
      "正常值    156864\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    789625\n",
      "异常值     10375\n",
      "Name: n8_outliers, dtype: int64\n",
      "n8_outliers\n",
      "异常值      2131\n",
      "正常值    157479\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    786384\n",
      "异常值     13616\n",
      "Name: n9_outliers, dtype: int64\n",
      "n9_outliers\n",
      "异常值      3953\n",
      "正常值    155657\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788979\n",
      "异常值     11021\n",
      "Name: n10_outliers, dtype: int64\n",
      "n10_outliers\n",
      "异常值      2639\n",
      "正常值    156971\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    799434\n",
      "异常值       566\n",
      "Name: n11_outliers, dtype: int64\n",
      "n11_outliers\n",
      "异常值       112\n",
      "正常值    159498\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    797585\n",
      "异常值      2415\n",
      "Name: n12_outliers, dtype: int64\n",
      "n12_outliers\n",
      "异常值       545\n",
      "正常值    159065\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788907\n",
      "异常值     11093\n",
      "Name: n13_outliers, dtype: int64\n",
      "n13_outliers\n",
      "异常值      2482\n",
      "正常值    157128\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n",
      "正常值    788884\n",
      "异常值     11116\n",
      "Name: n14_outliers, dtype: int64\n",
      "n14_outliers\n",
      "异常值      3364\n",
      "正常值    156246\n",
      "Name: isDefault, dtype: int64\n",
      "**********\n"
     ]
    }
   ],
   "source": [
    "data_train = data_train.copy()\n",
    "for fea in numerical_fea:\n",
    "    data_train = find_outliers_by_3segama(data_train,fea)\n",
    "    print(data_train[fea+'_outliers'].value_counts())\n",
    "    print(data_train.groupby(fea+'_outliers')['isDefault'].sum())\n",
    "    print('*'*10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "#删除异常值\n",
    "for fea in numerical_fea:\n",
    "    data_train = data_train[data_train[fea+'_outliers']=='正常值']\n",
    "    data_train = data_train.reset_index(drop=True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征分箱:\n",
    "**目的：**\n",
    "\n",
    "特征分箱主要是为了降低变量的复杂性，减少变量噪音对模型的影响，提高自变量和因变量的相关度。从而使模型更加稳定。\n",
    "\n",
    "**对象：**\n",
    "\n",
    "将连续变量离散化  \n",
    "将多状态的离散变量合并成少状态\n",
    "\n",
    "**优点：**\n",
    "\n",
    "处理缺失值：当数据源可能存在缺失值，此时可以把null单独作为一个分箱。  \n",
    "处理异常值：当数据中存在离群点时，可以把其通过分箱离散化处理，从而提高变量的鲁棒性（抗干扰能力）。例如，age若出现200这种异常值，可分入“age > 60”这个分箱里，排除影响。   \n",
    "业务解释性：我们习惯于线性判断变量的作用，当x越来越大，y就越来越大。但实际x与y之间经常存在着非线性关系，此时可经过WOE变换。  \n",
    "\n",
    "**原则：**\n",
    "最小分箱占比不低于5%  \n",
    "箱内不能全部是好客户  \n",
    "连续箱单调  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "#固定宽位分箱\n",
    "# 通过除法映射到间隔均匀的分箱中，每个分箱的取值范围都是loanAmnt/1000\n",
    "data['loanAmnt_bin1'] = np.floor_divide(data['loanAmnt'], 1000)\n",
    "## 通过对数函数映射到指数宽度分箱\n",
    "data['loanAmnt_bin2'] = np.floor(np.log10(data['loanAmnt']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "#分位数分箱\n",
    "data['loanAmnt_bin3'] = pd.qcut(data['loanAmnt'], 10, labels=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in ['grade', 'subGrade']: \n",
    "    temp_dict = data_train.groupby([col])['isDefault'].agg(['mean']).reset_index().rename(columns={'mean': col + '_target_mean'})\n",
    "    temp_dict.index = temp_dict[col].values\n",
    "    temp_dict = temp_dict[col + '_target_mean'].to_dict()\n",
    "\n",
    "    data_train[col + '_target_mean'] = data_train[col].map(temp_dict)\n",
    "    data_test_a[col + '_target_mean'] = data_test_a[col].map(temp_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 其他衍生变量 mean 和 std\n",
    "for df in [data_train, data_test_a]:\n",
    "    for item in ['n0','n1','n2','n2.1','n4','n5','n6','n7','n8','n9','n10','n11','n12','n13','n14']:\n",
    "        df['grade_to_mean_' + item] = df['grade'] / df.groupby([item])['grade'].transform('mean')\n",
    "        df['grade_to_std_' + item] = df['grade'] / df.groupby([item])['grade'].transform('std')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label Encoding 完成\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#label-encode:subGrade,postCode,title\n",
    "# 高维类别特征需要进行转换\n",
    "for col in tqdm(['employmentTitle', 'postCode', 'title','subGrade']):\n",
    "    le = LabelEncoder()\n",
    "    le.fit(list(data_train[col].astype(str).values) + list(data_test_a[col].astype(str).values))\n",
    "    data_train[col] = le.transform(list(data_train[col].astype(str).values))\n",
    "    data_test_a[col] = le.transform(list(data_test_a[col].astype(str).values))\n",
    "print('Label Encoding 完成')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征选择的方法：\n",
    "\n",
    "#### 1.Filter\n",
    "方差选择法\n",
    "相关系数法（pearson 相关系数）\n",
    "卡方检验\n",
    "互信息法\n",
    "#### 2.Wrapper （RFE）\n",
    "递归特征消除法\n",
    "#### 3.Embedded\n",
    "基于惩罚项的特征选择法\n",
    "基于树模型的特征选择\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 方差选择法\n",
    "\n",
    "方差选择法中，先要计算各个特征的方差，然后根据设定的阈值，选择方差大于阈值的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "#其中参数threshold为方差的阈值\n",
    "VarianceThreshold(threshold=3).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 相关系数法\n",
    "\n",
    "Pearson 相关系数 皮尔森相关系数是一种最简单的，可以帮助理解特征和响应变量之间关系的方法，该方法衡量的是变量之间的线性相关性。 结果的取值区间为 [-1，1] ， -1 表示完全的负相关， +1表示完全的正相关，0 表示没有线性相关。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from scipy.stats import pearsonr\n",
    "#选择K个最好的特征，返回选择特征后的数据\n",
    "#第一个参数为计算评估特征是否好的函数，该函数输入特征矩阵和目标向量，\n",
    "#输出二元组（评分，P值）的数组，数组第i项为第i个特征的评分和P值。在此定义为计算相关系数\n",
    "#参数k为选择的特征个数\n",
    "\n",
    "SelectKBest(k=5).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 卡方检验\n",
    "\n",
    "经典的卡方检验是用于检验自变量对因变量的相关性。 假设自变量有N种取值，因变量有M种取值，考虑自变量等于i且因变量等于j的样本频数的观察值与期望的差距。 其统计量如下： χ2=∑(A−T)2T，其中A为实际值，T为理论值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "#参数k为选择的特征个数\n",
    "\n",
    "SelectKBest(chi2, k=5).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 互信息法\n",
    "\n",
    "经典的互信息也是评价自变量对因变量的相关性的。 在feature_selection库的SelectKBest类结合最大信息系数法可以用于选择特征，相关代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from minepy import MINE\n",
    "#由于MINE的设计不是函数式的，定义mic方法将其为函数式的，\n",
    "#返回一个二元组，二元组的第2项设置成固定的P值0.5\n",
    "def mic(x, y):\n",
    "    m = MINE()\n",
    "    m.compute_score(x, y)\n",
    "    return (m.mic(), 0.5)\n",
    "#参数k为选择的特征个数\n",
    "SelectKBest(lambda X, Y: array(map(lambda x:mic(x, Y), X.T)).T, k=2).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wrapper （Recursive feature elimination，RFE）\n",
    "递归特征消除法 递归消除特征法使用一个基模型来进行多轮训练，每轮训练后，消除若干权值系数的特征，再基于新的特征集进行下一轮训练。 在feature_selection库的RFE类可以用于选择特征，相关代码如下（以逻辑回归为例）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "#递归特征消除法，返回特征选择后的数据\n",
    "#参数estimator为基模型\n",
    "#参数n_features_to_select为选择的特征个数\n",
    "\n",
    "RFE(estimator=LogisticRegression(), n_features_to_select=2).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Embedded\n",
    "基于惩罚项的特征选择法 使用带惩罚项的基模型，除了筛选出特征外，同时也进行了降维。 在feature_selection库的SelectFromModel类结合逻辑回归模型可以用于选择特征，相关代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "#带L1惩罚项的逻辑回归作为基模型的特征选择\n",
    "\n",
    "SelectFromModel(LogisticRegression(penalty=\"l1\", C=0.1)).fit_transform(train,target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于树模型的特征选择 树模型中GBDT也可用来作为基模型进行特征选择。 在feature_selection库的SelectFromModel类结合GBDT模型可以用于选择特征，相关代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "#GBDT作为基模型的特征选择\n",
    "SelectFromModel(GradientBoostingClassifier()).fit_transform(train,target_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本数据集中我们删除非入模特征后，并对缺失值填充，然后用计算协方差的方式看一下特征间相关性，然后进行模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':'Correlation of Numeric Features with Price'}>"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAHaCAYAAADouYt/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAB7TklEQVR4nO3dd7xcRf3/8dc7PZCQEHoPhNCFCKFKBxX9IUWkoyDtC6IgigoWRCwgCoiISC+CSi8iHQkdKSG00HvoLSG93c/vj5lNNps9c87d3Xvv5u7nmcd95O6ZM+fMtjtn5szMR2aGc84557qnHl1dAOecc851HK/onXPOuW7MK3rnnHOuG/OK3jnnnOvGvKJ3zjnnujGv6J1zzrluzCt655xzrhNI2kHSC5JelnRslfQVJd0t6QlJT0n6akPO6/PonXPOuY4lqSfwIvBFYBzwKLC3mY0t2+dc4AkzO1vSWsDNZja03nN7i94555zreBsBL5vZq2Y2A/gXsHPFPgYsEn8fBLzTiBP3asRBnHPOOZe0HPBW2eNxwMYV+5wA3C7pe8DCwPaNOLFX9M4551rWzI9ebdj96z5LDPs/4NCyTeea2bntOMTewMVmdqqkTYG/S1rHzNrqKZdX9M4551wDxEo9q2J/G1ih7PHycVu5g4Ad4rEektQPWBz4oJ5y+T1655xzrattduN+0h4FhktaWVIfYC/gxop93gS2A5C0JtAP+LDep+gteuecc62rvl7x4qcxmyXpu8BtQE/gQjN7VtKJwGNmdiPwQ+A8SUcTBuYdYA2YGufT65xzzrWsme+/0LBKsPdSq6tRx2okb9E755xrXW2d06LvSl7RO+eca1l1DmhfIPhgPOecc64b8xa9c8651uVd984551w35l33zjnnnFuQeYveOedc68pf6GaB5xW9c8651uVd984555xbkHmL3jnnXOvyUffOOedc9+UL5rimIWlTSVdKekfSDEkfS7pD0v6SenZ1+UokvS7p4hry7SLpB1W2by3JJG3dgOI1nKSFJf1d0gexnH9K7Gvx58AqaZdJer0jy9pRJJ0gqSHrhZe9RpU/lzXi+BXnGhzLvn6jj92MJI2SNKrs8Yj4/IdU2dck/abG85xQ8d6Nl/SIpH3bk7+Wc7vqvEW/AJD0feA04L/AT4A3gEWBLwFnA+OBG7qoeI2yC7A94XmWGw1sCozt7AIVdASwN3Ag8CLwboE8v5R0mZnN6NCSdZ7zgVsbeLyLgXMqttUdqrOKwcAvgXGEz1l3952KxyMIz/8y4JMOON/mwGxgCHAIcJmkvmZ2YU6+Rn+e0rzr3nU1SVsSKr+/mNmRFck3SDoNWLgB5+lrZtOrbO8NzGpEqMRamNlnwMNdce6C1gTeMbNLC+5/O+EC7f+AMzusVJ2g9Jkxs3GEyrJR3jazZn7Pk7K+S13NzDr7Yvl/ZjYLQNLtwHPA94GqFX0Hfp7SvOveNYGfEK62f1wt0cxeMbOnSo8lbSTpTkmTJE2WdJekjcrzSLpY0rh4O+BBSVOBUyQNjV1t35F0iqR3gOmElg+Svi7pYUlTYnfcVZJWTBVe0hKSzpH0Ysz3lqR/SFquvDzA/sByZd19r8e0+bruFRwt6YV4G+NdSX+RtEjFuU3SbyQdKek1SRMl3SNp7bwXPebfT9KTkqZJ+ih20S9TfnzgAGCFsnJvnXG4kkeB64GfSVooce7Se3FAxfZqr8coSfdL2kHSGElTJT0haWNJvST9Lr5Gn8T3fuGKYy4k6ffxNZoR//+ZpB5Vzvt1SedJ+hB4P6bN19Uaz/sTSWPj6/ehpFslrZHz+uQq8jmUtJek/8bzToqvx/7lry/wWnx4Xtn7d0BMr3oLKu5zQtnjUjf1OpJukzQJuDKmFXldB0g6U9KbkqYr3AK6M/U6xf1frtj2eCzHqmXbfivpfUmKj+d03cfneVHc9aWy5z+04rg1fXcqxQr/CWDVeNyGf54U/tb8TdLb8bV8XtKhtZS3u/EWfRNTuPe+DXC9mU0rsP+6wD2Ebu4DAAOOBe6RtImZPVm2+yDgX8AfgZ8CU8vSfkaokA4FegLTJB1GuE1wEXAiMBA4IR57XTObmFGsIcA04DhC9+uywA+BByStEZ/Xr4ElgA2BnWK+VIvot/F4ZwH/BtaKx1hP0lY27+ia/YAXgKOAPsAfCD0ha5RaG9XEPxDnAFfEcy0L/A7YWNL6ZjaJcEvhBGA9YNeYtUir6efAU8CRwMkF9i9iVcJz+y0wCTgFuDH+9CJ8HtaM+3xAvHCU1Au4jbmv4dPAJsAvCO/dDyvOcyZwC/BNoF+iPP8i3I75E3Bn3HdLYBng+ZznoliuOcpahkU/h6sAVxNe37Z47vMl9TezvxFusXwduBY4ifA6AbySU7YsNwAXAL8H2trxup5O+Mz/FHgJWAz4AvHiOsPdwHclrWhmb0palNANPxXYFihdBGwLjMrojfsP8BvCZ3F35ragy2891fTdSViZcJuxXEM+TwoX+fcD/Qmfh9eALwNnK/QUZPeetcCCOZiZ/zTpD7AUobI+qeD+VxO+SIPLti1C6BG4tmzbxfG4O1fkHxq3jwZUtn0AMAG4sGL/lYEZwPfLtr0OXJwoY09ghXieXSvKNK7K/lvHfbeOj4cQLgIurthvv7jfTmXbjPDHs3fZtm/E7ZvllPF94O6K7ZvHvEeWbbsMeL3g+2PAb+Lvf4/vy6Bqxyl7Lw5IvR5x2yhgJrBK2bad4n53VuS/Fnit7PE3435bVuz3s/jeLllx3uuqPK8Twp+SOY+3rXyd2vGZt4yfVdvzOaxI70G42DkPeLLKa3xwlTxVP8dx/xMqnztwVMV+RV/XZ4DT2vkaDSFcvOwfH+8CfEq40Pinzf3OzgQOq/icjCp7fEDptc14nu3+7lS8Jn3j674kYSyAAX/qiM8T4QJqGjC8Yvt5wEdAr6y808b+1xr1097Pe2f9eNd997IlcJOZjS9tsHCP+0Zgq4p9ZwI3ZRzneovfkmhTwgXD5bELrVdssbxFaJ1tmSqUpMMVusAnAbOAN2PS6sWe1jw2IbQuKkdh/yseu/J53mFmM8sePx3/T91yWJ3wx+ny8o1mdj9hIGTlOWrxS8If4x814FgAL5rZq2WPS63m2yr2ex5YvtSdC+xAeE4PVry3twO9Ca93uesKlOVLhD/M57XnCZS5kNC7U/7zFu34HEoaLumfkt4mfNZnAgdT22euiMrXpejr+ihwgKSfShqpAjNozOwT4ElCBUj8/x5CS3ebuG1LQiV7dx3PqZbvTrlphNf9fUKPxZ8IPYzlGvV52gH4H/Baxet9G6GXZK2CZe6WvOu+uX1M6I5bqeD+Q6g+6vs9wij9ch+aWVafVeUxloz/35mx/6dZBZL0PeDPhAGFP4r79iAMsEt11WUpTQWap4xmNkvSx2XpJZWjiUu3BFLnrnqO6L0q52g3M3tV0gXAUZLOqPd4zP8ezEhs70XotZhFeG9XIvxBrmaxisdFZhUsBnxiZlNz96zuXTN7rHKjpEKfQ0kDgDuAKYSK5RXC8z6cMDuiI1T7zhR5Xb9H+EwdSLjt8omkS4GfmdmUxPnuJrSwIVTu58dtS0laK257x8xeaO8TKVPLd6fcJoRR958Cb1ZcNJQ06vO0JKHXp+jneC4fde+6Uqy8RgFfVLGRvJ8AS1fZvjTz/8Gvdt8uK+3j+P8BwLNV9s+6Pw+wF3CXmc251ytp5cT+eUp/fJYuL0u8el+MxkwTKj9HpaWBxxtwDgj3bvcntHYqlcZk9KnYnv0HqzYfE+5n7pGR/nrF49TnpuQjYEi8H15rZV9N0c/hpoRKdovYCwPM+YwUNY2K115S6rWv9p3JfV0tjPU4DjhO0kqEyvtkwoXJTxLnuxs4WtJmwNrAf83sPUnPEVr421Jfa74RHrf8e/mN+jx9TBh7clRGevYFTwuMuveKvvmdTLi3dgpVPsSx0hxoYeT9PcBXJQ20OChJ0kDga/EYtXqQ8Ed0VTO7pJ15FwI+q9j27Sr7TScMpMnzMOGP4F7AXWXb9yR8nke1s3zVvEDobtyLcN8TgPhHdSXg1AacAzN7R9JZhFZd5XSy9wmvyToV2/9fI85d5lZgN2CSmeUNkivqdkJL+mAaO4Ww6OewNJthTusuDljbuWK/0oVztc/dG9T32rf7dTWzN4BTFRaWqTx3pXsIreUTCRXhM3H7fwmDDEcAf805Rur5N5Min6dbCd+jN83sg84q2ILCK/omZ2b3KqwYd1rskruYcI97UWA7wod/H8Io7l8DOwJ3Sfo94Wr5J4Q/fCfWUYbPJP0IOEvSEoRRshOA5Qj3q0eZ2T8yst8K/ETST4FHCC2Nb1TZbyzhqv1w4DFgmpk9XbmTmX0i6VRCC2gycDNhNPlvCKNu/1Pr8yw7x2xJxwPnKKzIdhnhuf6WMEApb8GP9jiZMLthK0LlUiqDSboCOEjSi4SLj/9HGMTUSJcTLrzuiq/rk4SW7DDCgL5dcrqQ52Nmd0u6hvCZXYFQ+fQm3Df+j5mNqqWg7fgcPki4uDxL0i8J60z8nFAhDio75PuEluBekp4CJhMGKn5MGPNxoaTTCWNZ1iP0JBRV6HWV9BBhDM3ThNkSW8VzJS+o42sxmvA34KqyMTV3ExZxgvC6p5RmiBwh6RLChdFT1mQLORX8PJ1OuNi/L75nLxDe9zUIPTuVF3lzede9awZm9idJjwBHE6bDLU5o2TxGWHjl33G/pxTmV/+W8IdChJbiVjbv1LpaynCOpLcI99n3IXx23gbuA8Yksp5ImCp0NOHe3j2EaS+vVux3PuGe3u/i/m8QRkVX8zPCVL3DCKt9fQxcChxnDVq42szOlTSF8HxvIPwRvhn4sZlNbsQ54nk+Vlj06IQqyUcRxjOcEP+/ktBqyRpEWcv5Z0r6MqHFdChhBPtkwn3t/zD3Xn977UW4yNyfsEjKBMLAs/PrLG/u59DMPpS0K6Hn5WrgHeAMwtiKX5Ydq03SwYTP3J3xWN8mXExfQpgdchDhO3YfYQrlPPPXE+Us+rreS+jePzae/1XgaDP7c4HT3E0YqPjfim1GaNm+llPGJxXWBDiUsHJdj1jO1wucu7MlP09mNiH2uB0f91uOMAPpBeCa1IGzhyp1H5p3cLVzzjnXOqY9eXPDKsF+631V+Xt1Pm/RO+eca10+GM8555zrxlrgHr0vmOOcc851Y96id84517q8694555zrxlogqI1X9E1I0oNmtlmV7RcT1rK/OpV/5kevZo4i/fP6x9dfQNfl8oYJN+XQ3wJ++GZlCIO5Tl1xv5rzF8nr2if1XqXU+1788M3LFtSPd5fxir4JVavknXPOdQDvunddQdIkMxsQI4ydCXyREKGrqVascs65BZ6PunddbFdCWM21gG8B3tJ3zjnXLl7RN7ctgX+a2Wwze4fE2tWSDpX0mKTHzr/0n51XQuecW5BZW+N+mpR33XcTZnYucC6kB+M555wr4133rovdC+wpqaekZYBturpAzjnnFizeom9u1xHCuo4lhKZ9qEim1BS6I0dnR6ud9vPvJI/78HWDkulbP3tSZtqpG6Sn9T1DdiTUcw5ZKDMN4LULxyfTT2vrl5m2mmWnQf6V8CKJxsD0nElAq06vff7uMn3SkWMv6ZkdYnx2zuS8k/ecmUy//+I+mWlDB32WzHvRjPRnKOXlHrWPRZ2o9HPeqW1SZtr4GX1rPi/AEgtNzUzLiyn26rSB6WMnxuf26pFuqU6enf3nf2Cv9GegHrNzvheTct6rhmuBFr1X9E3IzAbE/w34bhcXxznnuq1WCFPrXffOOedcN+Yteuecc63Lu+6dc865bqyJp8U1infdO+ecc92Yt+idc861Lu+6d84557qxFui694q+g0gaDOxjZn/t6rKUS82V7/ebdFE36XNkMv2RdX6cndg3PV99iHpnpo06Mz39pQ8LJ9MH9c+euLtwznd8Vs6c3wGJok3K+Xa92LdneoeESTMHJNOX7pFd8LwYnw9dnE7/uGf2E2ubsEgy79T+tf9RXYTaX69+Oc/6Ccuer554uoW8MSN73YG8Z7RQzpzyl3pkf6965UxHn5V4Xv3bsr+PJRvm7lFdj5xyLd3WyVFoW6BF7/foO85gIL0CTRkF/n4455xrKK9YOs7JwDBJYyT9QdKPJD0q6SlJvwKQNFTSC5IuBZ4BtpD0vKSLJb0o6XJJ20t6QNJLkjbq0mfknHPdTQsEtfGKvuMcC7xiZiOAO4DhwEbACGADSVvG/YYDfzWztYE3gFWBU4E14s8+wObAMcBPO7H8zjnX/bW1Ne6nSXlF3zm+FH+eAEYTKvDhMe0NM3u4bN/XzOxpM2sDngXuikvhPg0MzTpBeZjahye91BHPwTnn3ALIB+N1DgEnmdk582yUhgKTK/adXvZ7W9njNhLvV3mY2lNX3M/D1DrnXBFN3BJvFG/Rd5yJQGko723AgZIGAEhaTtKSXVYy55xzQQvco/cWfQcxs4/jILpngFuAfwAPSQKYBOwHdHrYpFSo2bzpc/2O/3MyffIV2UMIeuf0Maw/K3s6zzX9pyXz/mLQhGT6zAlDMtPyZvLkfXVXV2WHzFxT29LT/o7+5IGco2c7bonNkunLzcpOy+vuebhf+s/CPgM/zEw7Z1L2aw2wY3bE1lz1dFMtkfNNm5L4HCwzu74Oskf7Zn+KpuZ8wpa19DS3VMk+VfpJbzg9u533fPaMwLoNyHk5N9bEjjt5i/KKvgOZ2T4Vm86osts6Zfu/XvH4gKw055xzDeBd984551w31old95J2iFOqX5Z0bMY+e0gaK+lZSf9oxFP0Fr1zzjnXwST1BM4CvgiMAx6VdKOZjS3bZzhwHPAFM/u0UWO5vKJ3zjnXujqv634j4GUzexVA0r+AnYGxZfscApxlZp8CmNkHjTixd90755xrXQ3sui9fzyT+HFp2puWAt8oej4vbyq0GrBYHcj8saYdGPEVv0TvnnHMNUL6eSY16ERZT2xpYHrhX0ufMbHw95fKK3jnnXOvqvK77t4EVyh4vH7eVGwf8z8xmAq9JepFQ8T9az4m9om8xWz97UmZaMsws6XnyANs8+7vMtNEbHJ/M+0Dv6ZlpZ+6bnuz+2j/7JtOnJJYrmJIIjwv5YURHK3uu/MycOfp/G/yFnKNnGz5zSjL9ij7ZE6Fn5cxI/+0u6WM/+s/stRj26Ze9rgDALf3SYWy/lEibWMeyE+/0TD/nHW1S9nmtvknle1v2Z7stZyGHl1KxZIFlE8fOuy87OxG6d7kZHVf5Tcr5XtysdAjmzzewLEBnVvSPAsMlrUyo4PcixDIpdz2wN3CRpMUJXfmv1nvibnuPXtKDNebbRdJaBfY7QdIx8feLJX2jlvO1o1wHSFq2I8/hnHOuY5jZLOC7hJVSnwOuNLNnJZ0oaae4223Ax5LGAncDPzKzj+s9d7dt0ZtZeumwbLsANzHvSMhmcAAhlO07XVwO55zrPqzzQoOY2c3AzRXbji/73YAfxJ+G6c4t+knx/60ljZJ0dYz1frniOrSSTo4LEzwl6Y+SNgN2Av4Q48gPk3RIjCP/pKRrJC2Uc97XJZ0U8z8maX1Jt0l6RdJhZftlxad/TtJ5cbGE2yX1j70FI4HL43H7d9Tr5pxzLcXD1HYbnwe+D6wFrAJ8QdJiwK7A2ma2LvAbM3sQuJHQXTLCzF4BrjWzDc1sPUJ3y0EFzvdmjEN/H3Ax8A1gE6BUoX+JdHz6s2J8+vHAbmZ2NfAYsG8s13wrhnuYWuecc9W0SkX/iJmNizHexxDiuk8ApgEXSPo6kDUCaR1J90l6GtgXWLvA+W6M/z9NGEE50cw+BKZLGkw6Pv1rZjYm/v44iRj05czsXDMbaWYjNxkwPD+Dc865lmjRd9t79BXKh6bOBnqZ2SxJGwHbEVrc3wW2rZL3YmAXM3tS0gGE+Y1Fz1ceT770uBfp+PSVZfVueuec6yhNHF62UVqlRT+fGBt+UBwccTSwXkwqjyNP/P1dSb0JLfpGqCU+fWW5nHPOuVyt0qKvZiBwg6R+hBZ2aZTjv4DzJB1JaOn/Avgf8GH8v+7K1sxul7Qm7YtPfzHwN0lTgU2r3acv4tTUfPa+/ZJ582LKp+bK//DxE5N5//b57Lx/uCx94mVmp+fRr524nM270s271p+dmBM8LWe+8DvpKfxJ71lyTCgr1z7lnHOuzZ4nD6DUy50z57yeGenD2mrPPcDSb8aoxNe6R52x2WdZ9gvWK+czMjXn3O9Z9ne2R873NTWFf3pOuQBqndaUtzZF35z3quGauMu9UbptRW9mA+L/o4BRZdu/W7bbRlXyPUAYtFdydvyp3O+Est8PKPt9aNnvFxMq6GppZ5Afn/6PZb9fA1xTZX/nnHO16sTpdV2lZbvunXPOuVbQbVv0zjnnXC7vunfOOee6sRao6L3r3jnnnOvGvEXvnHOudbXAPHqv6FvMM5kLAMKQnJCt689Kp6dCzaamzwEc9kT29Lud1/9uZhrAdj2GJNMHJ77Hb+eEL82bCtQnMRVoktLHnqLa/8B8qlnJ9NXb0lMOU0YrHWp21US4hyfbJiTzbm+DaykSAHXMRswNcDs98V5NzXkf86Smi03K+QxMywkpvGjiEzqhR/pZD0zkfZ8Zybz1yHsvPsubF9hg1uaj7jtEDN7yTCeeb5CkSyW9HIPLXCopPVm448u0taSbMtLOLxIq1znnnMvTKvfoLwBeNbNVzWwY8BpwfmecWFK7e03M7GAza7Ywuc451/20wFr3XVnR96wSjnWEpIdj6NbrJC0KEMPMnh6jsz0naUNJ10p6SdJvSgeUtJ+kR2Io13Mk9ZS0KrAB8Ouyc58IjIxhaM+StFPMf52kC+PvB0r6bVbo2LjPMEm3Sno8Br5ZI26/WNLfJP0POEXSVrFMYyQ9Iam0DNeAjPC5oySNjL9Pis/9WUl3SVqiI98U55xrKdbWuJ8m1ZUV/XzhWIFLgZ/EsLFPA78s23+GmY0E/gbcABxBWEXuAEmLxSVl9wS+EEPEziasTb8WMMbM5twair+PIUSiuw/YIiYtx9xV8bYA7k2UFeBc4HtmtgFwDPDXsvIuD2xmZj+IaUfEcm0BlJavnS98bpXXaWHgsXjueypekzk8TK1zzrlqurKirwzHOgwYbGb3xG2XAFuW7V8e+vVZM3vXzKYDrwIrEKLQbQA8KmlMfLxKgXLcB2wR74mPBd6XtAywKfBgRlmHxoA0mwFXxfOdAyxTdtyryi4uHgBOi+vnDzaz0kiqauFzK7UBV8TfLwM2r/YkPEytc87VoM0a99OkunLUfWU41sEF90+Ffr3EzI4rzxS77kdI6hErVCT1AEYAY83s7RgjfgdCC34IsAcwycwmSlqsSln7Ey6SxsdWejVzhi6b2cmS/gN8FXhA0pczXoMi70fzfpqcc25B08T31hulmQbjTQA+lVTqRv8moau6qLuAb5TCvUoaImklM3sZeAL4edm+PwdGxzSAhwld6PcSWvjHxP8zmdlnwGuSdo/nk6T1qu0raZiZPW1mvwceBdZox/PqQYiiB7APcH878jrnnGtxzTaPfn9CKNaFCF3y3y6a0czGSvo5cHtssc8k3Md/AzgIOFPSK3H3h+K2kvuAL5nZy5LeILTqkxV9tC9wdjxvb0KI2yer7Pd9SdsQeh+eBW4h3BooYjKwUTzHB4RxCDU755DsOdCjzkzPcL2m/7Rk+pn7Zs8Xzgs1m5orf8PovyTzXpAzR//jxGT4WXkRMXP6T663DzLTDpy5ZDLv4DpCya6Q815cr+z4ppNz5m6fs0M6AvKNNy6cmXboCtnrNAD8+f3aozzX0+4alJN52MyZmWmf9KhnBj8sS/Z71adn+kNgOSFb35ndPzNtkKWP/VGP7C/GsN4dN4++b853arUZHqa20bqkojez18kIxwpsUmX/rct+H8W8YWfL065g7v3s8vyfEuK9Z5XnAsIUPMxsJmEAXG5Zzew1Qpd/5fEOqHj8vSqnrXwe3y37fevyHeOAPuecc43mYWqdc845tyBrtq57V8HMBnR1GZxzrtvyrnvnnHOuG2viaXGN4l33zjnnXDfmLXrnnHOtq4mXrm0Ur+hbzGsXjs9M60P2tCmAXwxKhyB97Z/ZoVGXmZ0Om5oKNZs3fe6gRIhbgBNG/jwzbUhbulMrr1PviFmLZ6a9kT3DDYDb+9Q+hWltSwdfXDLxt2tWTvDd225YLJm+RFt2iNxr3l02mXetOqYU5oX9TZmS03f5Yp/0m9W7jt7dt3pkT4HL61LNnvQXKPFW9iD9nRuQ+IzcpvxpkJ/P3aM2PTt7TTDvuu86km6OK9Yh6cgYWOZySTtJOrbGY/604vGDWfsWONaOMUDNk5LGSvq/uH2XIiFmi+7nnOta9VTyzjWDpmvRxwhuMrOvlm3+DrC9mY2Lj2+cP2chPwV+V3pgZpvVWMbehIA2G5nZOEl9mbtO/S7ATYR181OK7uecc66DWAuMuu+wFr2kkyUdUfb4BEnHSPqRpEdjKNpfxbShkl6QdCnwDLCCpNclLS7pb4TgNLdIOlrSAZL+EvMtFUPLPhl/Novbr4+hY5+VdGipPED/GCr28rhtUvxfkv4g6RlJT0vaM27fOoaMrQwlO5BwkfQxgJlNN7MX4vl3Av4QzzNM0iHx+T4p6RpJC2XsVx6adnFJr8ff19bc0LtPSfKINc451ygtENSmI7vuryAEhynZA/iQEPJ1I0JQmQ0klSLUDQf+amZrm9kbpUxmdhjwDrCNmZ1ecY4/A/eY2XrA+oTlZQEOjKFjRwJHSlrMzI4FpprZCDPbt+I4X4/lWQ/YnlABlyLRzRdK1sw+IfQqvCHpn5L2jUFzHozbfxTP8wpwrZltGMv4HHBQxn5ZDgPOiMFzRgLjEvs655xz8+iwit7MngCWlLRsDPbyKfA54EuEIDOjCcFdSi3UN8zs4XaeZlvg7Hi+2WZWGi12pKQnCcFqVig7R5bNgX/GY7xPCKazYUyrGkrWzA4mhMJ9hBAE58KMY68j6T5JTxPWxl+7nc/xIeCnkn4CrGRmVRcil8ejd8659rO2xv00qY4ejHcVIfLanoQWvoCTYit2hJmtGteZh7KwrvWQtDWhVb5pbEU/AfSr45CZoWRjRLrTgS8Cu2Xkvxj4rpl9DvhVoiyzmPt+zNnHzP5B6OafCtwsadtqmT0evXPO1cC77ut2BbAXobK/CrgNOFDSAABJyymGla3RXcDh8Vg9JQ0CBgGfmtkUSWswb5CcmXEgXaX7gD3jMZYAtiS01KuSNCBeUJSMIETJA5hIuIdfMhB4N563/JZB5X6vAxvE30thaZG0CvCqmf0ZuAFYN6tczjnn2qmtrXE/TapDR92b2bOSBgJvm9m7hApvTeChMKaNSYSocrXOrj0KOFfSQfEYhwO3AodJeg54gdB9X3Iu8JSk0RX36a8jhI19kjB1+sdm9l68UKhGwI8lnUNoaU8GDohp/wLOk3QkocL+BfA/wviE/zG3cq/c74/AlXHw4H/KzrUH8E1JM4H3KJs1UIvT2rI7Nwb1T4eHnDkhe647wJTE27h2ziXl4MR3JBVmFtLz5AFOeOw3mWlnrJ+eo5/n4X7ZV/FDci7wt5yVPb86z2u90n9U2nrUfg3/ap/052BKYh7+zJwQuNNVe7mOffzXNeedkhP5dGoifVLOc8ozOLFWw/Sccs2oY055Xqk/7pV97B50XKjYvOf8Un1RgV0VHT69LnZZlz8+Azijyq7rVOw3NOP3iwnd4cT76TtXOdZXMsryE+AnZY8HxP8N+FH8Kd9/FBmhZIHy6X/leR4gDNwrOTv+5O0H87bWfx73Oxk4udq5nHPO1amJu9wbpenm0TvnnHOdpokH0TVK066M55xzzrn6eYveOedc6/Kue+ecc6778iVwnXPOObdA8xa9c8651uVd964ISbsAL5rZ2LJtxwAHA9MIYaXPNLNLCx5va+AYM9ux0WVdzbLn0S+c04PVljP/dUrVtYiCvK6jt3tmf9lm5Zw3L6Z8aq78UaPTseyv/9wvkumL9syeUz4+50nnpSfPm/OcU1/svI7KmTmv9yDL3qF/zodkUh3Ts4useXDMm5dV3Z4Xajb1nBex+jo+U8fOWSKCZWanX7BPU/Hoc55zW2KufOLrWLe8j8DAxOerQ7RARe9d942xC2Vz4iUdRlgWd6MYjGY78j/fzjnnXMO1fEUfQ+SWQtA+F0PSLiRpO0lPxLC1F8aY86Xwu2NjyNg/Vgs5S4h7f7iZfQZgZp+Z2SUxf9Zxd4jlGE2Iplcq38Jxv0divmoLBDnnnKuFB7VpGasTQuSuCXwG/ICw+t6ecWW/XsDhkhYDdgXWNrN1gd9UhpwlLHU70MxerTyJpH4Zx+0HnAd8jbDe/dJl2X4G/NfMNgK2IVxQLNzg5++cc63Jg9q0jLfikrQAlxG62l8zsxfjtksIgW4mEO65XyDp68CUdp5n9YzjrhG3vxSX4y2/0fgl4FhJYwjL8fYDVqw8sIepdc45V41X9EHlpdj4qjuZzQI2Aq4GdiQE0Knc5zNgUow61wgCdisL7buimT1X5bweptY559rJ2qxhP83KK/pgRUmbxt/3AR4DhkpaNW77JnBPDK87yMxuBo4G1ovplSFnTwLOkrQIzAlr+y1CNL35jgs8H7cPi9v3LjvWbcD3FMP9Sfp8Q56xc865lui69+l1wQvAEZIuBMYCRxLC214lqRfwKPA3YAhwQ7ynLsK9fJg/5OzZwADg0RhediZwqplNk/TtyuOa2fRSeFpJU4D7mHvh8GvgT4Twuj2A1wi9CTVJXdnlTWPLG2qSmipUT968SJ31fL3yps/t8nQ6NOoFn8+e8pV3FV3P0J16nnO95Uqlz8rL26RzT+r57NZz7LwpcDmz69JlqyOvtwC7F6/og1lmtl/FtruAytbzu4Su+3lkhJw9Jf5U7lvtuJjZrYR79ZXbpwL/lyq8c865GvkSuM4551w31old93Ea9QuSXpZ0bGK/3SSZpJGNeIotX9Gb2etmtk5Xl8M551z3JakncBbwFUIP8N6SKnuCkTQQOAr4X6PO3fIVvXPOuRbWeS36jYCXzexVM5tBGNtVbQG0XwO/J0zlbgiv6J1zzrUsM2vYT47lgLfKHo+L2+aQtD6wgpn9p5HP0St655xzrgHKFy6LP4e2I28P4DTgh40ul4+6d84517oaOP/dzM4Fzs1IfhtYoezx8nFbyUBgHWBUXDZlaeBGSTuZ2WP1lMsr+iYk6QRgEvARcLuZvRO3nw+cVh4Ot70WScwkGTA7nXd1TU6mj04swZ83H7hPIjTl9fZBMu8RsxZPpj/cL/uLnAozC+l58gAHPZEd5jYvrOrHPWqf1vP9ld9Opp/96vKZaXmhUXftOSGZPnlan8y0RQelV4W+fkr6vUp5u0feLP1sk5T+Y77t9BmZacsO+azm8wI89climWlTe6Q7VTdb4v1k+gcfDsxM+2x2dthogJUXG5+ZtvCQ7NejXnl/C1aa0ckLz3TeQjePAsMlrUyo4PciLNAGgJlNAOZ8QSSNIoQrr6uSB++6b3YHAMuWHpjZwfVU8s4557pGXEL9u4TVTp8DrjSzZyWdKGmnjjy3t+ibhKSfAfsDHxAGbDwOjAQulzQV2BS4hQZd4TnnnKNT16iPy6ffXLGtatefmW3dqPN6Rd8EJG1A6MYZQXhPRhMq+scoq9jjfRvnnHON0sRr1DeKd903hy2A68xsSox+d2N7D+Bhap1zzlXjFX034WFqnXOuBm0N/GlSXtE3h3uBXST1j8sffi1urwx/65xzroFaIR6936NvAmY2WtIVwJOEwXiPxqSLgb+VDcZzzjnn2sUr+iZhZr8Fflsl6Zqy37eu9zzTE+P5JuV8Gqa2Zc+TB5iZOPa0nHGEqXnOB85cMpn3jexp3QAMSVxoj8/p08rr8krNlT9qdPYce4A9N/h+ztGz3fLiCukdemU/6Sk5c8qvnT0omd6nd/abuean/dLl6ptOTlnSav9ztdm09CIRo/plf4gGTlqi5vMCTOmbWMchp7t31EdLJdOnJxZFWDRnSdZnEs9r0fRSCkCY+1uLw7/2STL99H8vWuORa9TELfFG8YreOedc62rie+uN4vfonXPOuW7MW/TOOedaVjMPomsUr+idc861Lu+6d84559yCzFv0zjnnWpZ33btuZ9Xp2dOMXuybDmB69CcPJNP/NvgLmWnvpCNmMkXZ/WeDc8Ln3t4nHVJzy1n9M9Pyptfl9eqlQs3mTZ+74vE/5Rw926Uj0iFw75mdHdq3t9Lv826WDiV7xAd3Z6adv8Q2ybyfJd7nPMvVHqWWP/YZn0xfWdnrUr1jU2s/MbD7rME1551QR5/r+J7pOa2jGJ+ZthuDaz9xjp/cNCCZ3lc5X/hG86775iHpSEnPSfpU0rE1HmO2pDGSnpH0b0mDG1zM1LkPkPSXzjqfc845BwtWi/47wPZmNq6OY0w1sxEAki4BjqD6IjXOOedagHmLvjlI+huwCnCLpKNLLWNJS0m6TtKT8WezuP0HsdX+jKTvZxz2IWC5uP9Gkh6S9ISkByWtHrcfIOlaSbdKeknSKWVlOkjSi5IekXReWZmWkHSNpEfjT3Z/dkZZJf1I0pHx99Ml/Tf+vq2ky2t9HZ1zzlXwoDbNwcwOA94BtgE+LUv6M3CPma0HrA88G2O7fxvYGNgEOETS58uPJ6knsB1zw8E+D2xhZp8Hjgd+V7b7CGBP4HPAnpJWkLQs8It4/C8Aa5TtfwZwupltCOwGnJ/1vBJlvY8QuhZgJDBAUu+47d6MY3mYWuecc/NZkLruq9kW+BaAmc0GJkjanBDbfTKApGsJFeQTQH9JYwgt+eeAO+JxBgGXSBoOGFA+dOwuM5sQjzUWWAlYnHCB8UncfhWwWtx/e2Atac5AmEUkZY0+ySrr2cAGkhYBpgOjCRX+FsCR1Q5kZucC5wKcuuJ+3X8YqXPONYB33Xc/pXv0KwEi3KMH+DVwt5mtQwgRWx6ZY3rZ77PJvzjqAWxiZiPiz3JmNqk9hTSzmcBrhLgRDxJa+NsAqxIuUJxzzjWCd903vbuAwyF0x0saRKgUd5G0kKSFgV3jtjnMbAqhZfxDSb0ILfq3Y/IBBc77KLCVpEVj/t3K0m4Hvld6IGlE4jipst4HHEPoqr8POAx4wiwnJJVzzjlXZkGv6I8CtpH0NPA4sJaZjSbEcX8E+B9wvpk9UZkxbnsK2Bs4BThJ0hMUuJ1hZm8T7uM/AjwAvA6UAjseCYyU9FTs6j+sLOsBksaVfgix57PKeh+wDPCQmb0PTKPigsU551x9rK1xP81qgblHb2ZD468Xxx9iBbhzlX1PA06rsn1AxeOvlT1crez3n8f0OeeKj3cs2+cfZnZubNFfB1wf9/mIMHiv8tzzHKtMVlnvomysgJmtVrmPc865+jRzBd0oC3qLviudEAf2PUO4n359l5bGOeecq2KBadE3GzM7pqvL4Jxzrj6t0KL3it4551zrsnRMgO7Au+6dc865bsxb9M4551qWd927BVJqov0yfaZkpk2amQ4fedwSmyXTh8/MPvZ7tlAy76fKjkG6Qv9pybxr26Bk+mu9sr/Ji7alO7XyFi34/spvZ6bd8uIKybx5oWZTvjXmxGT6BxtkHzuvo3KXldNxo+7ruXFm2l6XfTmZ9+T9bs05e7aXe9e+hMRXlA69e+gBqdCoA+mx7udqPjcTxmcmzR6bXq569kcTk+kL/f7szLRZo/6ZzPut3n0y03ptsUcybz2GtWWfF2C/VeqJW9Z+1uZd966DSPqTpLcl+XvgXBOrq5J3Ta8V5tF7JdMFYuW+K/AWsFUXF8c551w35hV9FZWhYyUNlfS8pMslPSfpakkLxX03kHSPpMcl3SZpmbh9lKTfxzC2L0raouwUWwPPEoLX7F123qywu9+KK+09KenvnfZCOOdcN2emhv00K6/oK1QLHQssCqwO/NXM1gQ+A74TQ8eeCXzDzDYALgR+W3a4Xma2EfB94Jdl2/cG/klYUe//xeNA9bC7axNW6ts2bj+q8c/aOedak3fdt6Y5oWNj1LlS6Ni3zOyBuM9lcb/VgXWAO+IqeT8Hli871rXx/8eBoQCS+gBfBa43s88Ia9yXRjBtS2jlY2azY3jcbYGr4tK6lELjVvJ49M4556rxUffFVQ75NcIA5mfNbNOMPKUQt+Xhbb8MDAaejjHrFwKmAjfVVbiyePR/9Hj0zjlXiI+6b01ZoWNXlFSq0PcB7gdeAJYobZfUO3a1p+wNHGxmQ2OgnpWBL8Z7/tXC7v4X2F3SYnH7kAY+V+eca2lmjftpVt6ir2BmoyVdTAgdC3A+8CmhUj9C0oXAWOBsM5sh6RvAn2Ol3Av4E2Gg3XxiZb4DZaFrzWyypPuBrxHuv58r6SBCL8DhZvaQpN8C90iaDTwBHJB6Dqnr00t69s9MW7pH+sp2ueyp7gBc0Sd7fuzKqWnKwOptfTPTrld63u2SOffG2npkX8/W+wU4+9XlsxN7pb/598z+oObzpubJAxzzeGKe/czp2WnAnpv8JJm+/JwhJfO7cJ/bknl719G06FvHH9K8KfgnXZz92V+0bWztJwYm9Mg++aJtCyfzzlQ6vf8Gv8pMWyjnOX/cMzttOk+nMwO/eOPy3H2qyXsbH3h+2WT612s6a2vzir6KyjC3koYCs8xsvyr7jgG2rLJ967LfPyLeowfma5GbWflnt1rY3UuASwoW3znnXEGt0HXvFb1zzrmW5RW9A8DMXieMrnfOOecWKF7RO+eca1nNPIiuUbyid84517Jaoevep9c555xz3Zi36J1zzrWsZl6jvlEaXtFLOgGYZGZ/rNg+FHiOMB+9D/AYcJCZzWzn8cuPI2Ay8G0ze6HG8r4OjCwtMdsokkYAy5rZzTn73QzsY2bjG3n+LLMTs1jzPu55t7Jm5e5Rm8lKT5SfRWJCcI685anzurxSZ56i9OvRW7WXO/dPU2qufO/sNQuKSL1m/XNe0Ak5azWk9Mh/1pk+ScxlB5iZ+OzOrrMeaEscu39OJZP3+UzlHjA7/Zw/6Nk1FdysJqtXm3mN+kbp7K77V8xsBPA5wprwe9RznBjk5RLgpw0q33wU1PI6jSCsaZ9kZl/trEreOedc68mtwLJCtEp6XdLicZ+RkkaVZVtP0kOSXpJ0SOUxzWw2YeW55WL+rFCvq0q6M4ZnHS1pWJUiLkJYua5U1vvivqPLwrxuHcPGXl32XOa5rpTUX9Itkg6Jx3lB0qXAM8AKkiaV7fuNuHoeki6W9LcYUOZFSTvGwDUnAntKGiNpT0kDJF0k6ekYcna3mL/8ddwvhrUdI+mcuAxuz3iOZ2Leo/PeM+ecc8W0mRr206yKdt2vTuhmfyAuAfudnP3XJYR4XRh4QtJ/yhMl9SOEgT2qLNTrzmb2oaQ9CaFeDwQuB042s+tinh7AksCwGC1uICEozMbx0B8AXzSzaZKGE0LBjoxpnwfWBt4BHgC+QFivHmAA8C/gUjO7NN4eGA7sb2YPxzKnnu9QYCNgGHA3sCpwPOGWwHdj/t8DE8zsc/HxohWvyZrAnsAXzGympL8C+xKW013OzNaJ+w1OFcQ551xxrXCPvmiXdLUQrSk3mNnUeN/7bkIlCHMr6PeBd83sKTJCvUoaSKjgrgMws2lmNiUep9R1P4wQ6/3cuL03cJ6kp4GrgLXKyvSImY0zszZgDHOXpAW4AbjIzC4t2/ZGqZIv4EozazOzl4BXgTWq7LM9cFbpgZl9WpG+HbAB8Gh8HbYDVonHW0XSmZJ2AD6rVgB5mFrnnHNVFG3RVwvROou5Fwr9CuwPsYKOXdUPSNoJeI0qoV5jRV/EjcBF8fejCRcR68WyTSvbr3x0UnnYWAgt/B0k/cNszvIJkxPPqejzbQ8Bl5jZcfMlSOsRwtseRhjXcGDlPuVhak/1MLXOOVeIz6Ofq1qI1tcJLVCA3Sr231lSvxhadWvg0fLE2NI/FjiOjFCvZjYRGCdpl7i9b4z+Vmlz4JX4+yBCT0Eb8E3Sg6LLHU+4z39WYp/3Ja0ZB+btWpG2u6QecQzBKvE5TSTcWii5Azii9KCy654QovYbkpaM6UMkrRQvinqY2TWE3o71Cz4n55xzOTxM7VzzhWglDKa7QNKvgVEV+z9F6LJfHPi1mb0T73uXux44gXB/PSvU6zeBcySdCMwEdifMOCndAhAwAzg4HvOvwDWSvgXcyvyt8pSjgAslnRKPU+lY4CbgQ8LUwAFlaW8SXo9FgMPiGIG7gWNjOU8CfgOcJekZQo/Cr4BrSwcws7GSfg7cHi8mZhIuDKYCF2nuyP/5WvztcfKe2bMZH7o4nffhfumPy293mZKZds61g5J5Ryv7rTpnh6nJvLfdsFgy/dU+2VfsM3Mu5vNm3uzac0Jm2rWz0895N1s85+jZdll5XDI9L9RsyhWP/ymZPvl7B2WmTXw53Xa47INlaikSUPyqvZq3NCOZftoOEzPTbGZOjOUcsz7JPvf0D9Ov162vLZdM/+pqb2WmzZySfsW+sm7293nyc+2a9dwu03I6PB/sl073MLXtV7Sirxai9T5gtcodzeyEageoDAwTu8jXK9ulWqjXl4BtqxyualD1uP+6ZZt+ErePouxipDRALv4+tGz/b5f9Pk8QGzO7Gri62nmBO83ssIr9PwE2rNhv/yplHlr2+xXAFVWO761455zrAK3Qde8r4znnnGtZzTwtrlFy79Gb2eulqV1ufmZ2QGztO+ecc5kk7RDXaHlZ0rFV0n8gaWxca+UuSSs14rwe1MY551zLMlPDflIk9SQM+P4KYer33pLWqtjtCcL6K+sSbhWf0ojn6BW9c865ltWJo+43Al42s1fNbAZhkbad5y2L3V22XszDhKXi6+YVvXPOOdcA5QuXxZ9Dy5KXA8qnSYyL27IcBNzSiHL5YDznnHMtq5GD8coXLquHpP0Iy7dvVXeh8Iq+08R1BG4ys3UkHUDZOvgxfRRwjJk9JumnZva7srQHzWyz8mPUWo77L+6TmfZxz/THYZ+BHybTH/1n9rxx5URGXbXqWkjBjTcunMy7RNusZPqUxAzsQXWGCZ08Lfv17NM7fewjPrg75+jZ7uu5cTJ9efXOTMt9Tol58gALn3lBZtrPR6aXedhiRu2rikyvIwzyipb9PgE8cN3gzLR6o5he1z97Hn6vnNC7a+f0uX7r5ezvzSI90s/5oNerzlIG4LJ+6XUHIKyFXouFcp7zg7MrVwfvWJ241v3bwAplj5eP2+YhaXvgZ8BWZpaIN12cd93nUNDZr9M8YXfNbLNOPr9zzrnGehQYLmnlGOF0L8IS7nNI+jxwDrCTmX3QqBN7RV9FlTC1v5D0aJzy8Ku4z8mSype0PUHSMfHC4A9lYWX3bOe5Twb6x1C1l8dtk3KyOeecq0FnDcYzs1nAd4HbgOcIwdCelXRijPsC8AfCqqtXxTrgxozDtYt33WcbTljJbhHCEr0bEZbcvVHSloQV7P7E3PXx9yAEnvk6MIKw6t/ihGh09xY9qZkdK+m7ZjaiIc/COedcps5cMMfMbgZurth2fNnv23fEeb1Fn60UpvZL8ecJYDQhBO1wM3sCWFLSsjG63Kdm9hYhyM4/zWy2mb0P3MP8S+FmXfvVfBPSw9Q655yrxlv02UpRVgScZGbnVNnnKkJrf2mqr1Gf5WOgMnrdEOCj9hayxMPUOudc+3XiYLwu4y36fLcBB0oaACBpuVIoWULlvhehsr8qbrsP2FNST0lLEIL1PFJxzEeBL0haOh5zJNCXuXMsZ0qJYdPOOecaos3UsJ9m5S36HGZ2u6Q1gYckAUwC9gM+iAMpBgJvm9m7Mct1wKbAk4Su+B+b2XvlYXrN7H1JRwE3xxH9k4C9zaw0k+dc4ClJo81s30Y/p6GDPqu6/eNJQ1h0dnqq2jmThmSm7cBshvSbVj0xZ3oTwJNt1UO+rs5gtlrhncx8Y19fkuf6pj7KxkxldHKY6J/o/5gs6JvTP7LooIzwvFMWYs3p6clZ5y+xTdXtB394d2YawJ1M4fzLds1M//E3b2KN2dWvFcf2nMkGM9PXkalwswsDRyem0Z3+2EmZadd+7heZYYNXqWPqXREjct6LFQdUD1X7+qSBbDzy3appAP97bJlk+nXPLslvh1UfQP3LV5bitFOyZ8te8MMXOejv1QJ4Bv/+5lVcd1T1NVf2P/NDLrszO1zx3Zv/mQ0zyn3ZM4tx3oU7ZOY95MBbM9OKWDQV+Vdw7mL1hQZ28/KKvooqIXXPAM7I2PdzFY8N+FH8SR3zBuCGjGP+hBhiNz4eUO0Ytciq5IG6Knkgu5IvIKuSB5KVPJBTyZNdyUOykoc6KnmouZLPSwOSlTyQWckDdVXyUHslD2RW8nlp9aq1kgeSlXiR9KxKHkhW8kCykgcyK3kgWckDmZU8kKzki6SnJCt5Or+Sb4X7nF7RO+eca1nN3OXeKH6P3jnnnOvGvEXvnHOuZbXCqHuv6J1zzrWsemMZLAi8694555zrxrxF75xzrmVZTjS97kCWtxK/W+CkVsb7oEf21JWpOZ1YO05NfyEe75d93dgn52OWSn+rZ3q6zVozs8PQAnyQuJztkVOutg78G/CZau807JVzXzE1ga5HnX/YUvPd86bIHfP4icn03ouvkpl2+bL7pQsG7PtO9eCpv14pvRxF/8Tr2a/OP5GfJj5ki+S8jxMSU0MBeifey7zvXGrq6MQCfb3HvVFboNpTVkq/j71zyn30m5c19Fs5aqndG1YJbv3+VU151dAUXfeSHiywz/mS1oq//7QirUj+SfH/oZKeqZJ+YowDnJV/l9L54+NNJP0vRhh6TtIJcfvWknLDyhbdzznnnKtHU3TdF4m3bmYHlz38KfC79uQvcPzjc3bZBbgJGBsfXwLsYWZPSuoJrB63b01Y6S7v4qPofs455zpIWwt03TdLi77U2t5a0ihJV0t6XtLliuvOxu0jU/HaJQ2QdJek0TEW/M7tKMPFkr4Rfz9Z0tgYf/6PseW9E/CHeN5hwJLAuwAxUt3YuMztYcDRcb8tJH0ttvyfkHSnpKUy9ptz/orntIyke+N+z0jaoq4X2znn3ByGGvbTrJqiRV/h88DawDvAA8AXgPtLiTnx2qcBu5rZZ5IWBx6WdKO1YyCCpMWAXYE1zMwkDTaz8ZJuBG4ys6vjfqcDL0gaBdwKXGJmr0v6GzDJzP4Y91sU2CQe62DC2vc/rLLfQRlF2ge4zcx+G3sOFsoo96HAoQDfWHQjNhkwvOhTds451401RYu+wiNmNi4GeBkDDG1HXgG/k/QUcCewHLBUO88/gXDBcIGkrwNVFzM3sxOBkcDthMo4K8rD8sBtkp4mrH+/djvL8yjw7TgG4HNmVnVRbjM718xGmtlIr+Sdc66Ytgb+NKtmrOinl/0+m/b1OuwLLAFsEFv87wP92nNyM5sFbARcDexIdgWOmb1iZmcD2wHrxd6ASmcCf4nBb/4vUZ5ZxPcjRrTrE89xLyHU7dvAxZK+1Z7n45xzLlsrdN03Y0VfRFa89kGE8LEzJW0DrNTeA8e484PM7GbgaGC9mDQRGFi23/8rjR8AhhMuSsZX7hfL9Hb8ff+y7ZX7vQ5sEH/fiThDStJKwPtmdh5wPrB+e5+Tc8651rWgVvSleO2XV2y/HBgZu8m/BTyfkX91SePKfnYvSxsI3BS7/+8HfhC3/wv4URxUNwz4JuEe/Rjg78C+ZjYb+Dewa2mQHXACcJWkx4GPys5Tud95wFaSniTEs58c99saeFLSE8CeZITLdc45136t0HXvC+Z0T/6mOue6q4b2kd+81F4N+3v51ff/1ZT998046t4555zrFM18b71RFtSue+ecc84V4C1655xzLasj41k0C6/onXPOtSxfAtc555xzCzRv0XdDp66YHQby5R4zMtMWIR3uNW9o6kSyw8kOa+uTzJsKq5o3bWVSTijPYx//dWbaGevnxTJKe7vHrMy0JS399VouO2uul3NieaZCkOaFqU1/CmB6HZM6VpmZPndWmFmAmR+9mnv8rDC3eWFq81o8U3I+YymDLPvoeSFZJ+acN/XdSH2nAKYmjp0XBhnghDcqZzcXkxemdlrO5+v4Gs+bpRWmKHmLvpNIGizpO/H3ZSWV1swfIemrZfsdIOkvXVVO59y86qnkXfNrhXn0XtF3nsHAdwDM7B0zK0WqGwF8NSOPc845Vxfvuu88JwPD4kp6LwFrEpazPZEQdndz4KTyDJKWAP4GrBg3fd/MHui0EjvnXDfXpu4/GM8r+s5zLLCOmY2I8ehvMrMZko4HRprZdyF03ZflOQM43czul7QicBvhAsE551wDtMKNGa/om9v2wFpzY+ewiKQBZjapckePR++cc64ar+ibWw9gEzOblrejmZ1LCPbDqSvu1woXqc45V7dmHkTXKD4Yr/NUhqXN2w5wO/C90gNJIxpfLOeca11tatxPs/KKvpOY2cfAA5KeAf5QlnQ3oXt+jKQ9K7IdSQi7+5SkscBhnVRc55xz3YSHqe2e/E11znVXDW07X75s42517vvOZU3Zrvd79M4551pWK7SKvOveOeec68a8Re+cc65lNfMgukbxit4551zL8ul1zjnnnFugeYveOedcy2qFwXhe0Tc5SUOAK4ChwOvAHmb2aSpPKh59Kr51v5xZK0tkh5sH4J2e2ccekBPfOnXoQTl9a1Ny+qWmJE6dFw88z6TE67nZtPQL9sc+42s+71e0eDI99bw+6ZF+0m9pRjJ9ReuTmTZievrNGtM3/Wb9IhFrPC+mfCp/Xiz7X438eWbawol48kV8quzPwTozeybzJr5SALydCDq/0ox05tf6ZH8xehX4Xhzz5mX5O1Xx3PB0sM6nJi2aTN/z3cbGo2+Fe/Tedd/8jgXuMrPhwF3xsXPOOVeIV/RNQtJQSc9JOk/Ss5Jul9Qf2Bm4JO52CbBLlxXSOee6mbYG/jQrr+iby3DgLDNbGxgP7AYsZWbvxvT3gKW6qGzOOdfteEXvOttrZjYm/v444b78HBbWK65690zSoZIek/TYw5Ne6tBCOuecW3B4Rd9cppf9PpswWPJ9ScsAxP8/qJbRzM41s5FmNtJj0TvnXDGmxv00K6/om9+NwP7x9/2BG7qwLM451620Qte9T69rficDV0o6CHgD2KOeg+3UNikz7QkbmMybmqYGsKNlH3sU6WNPT0xTGzZzZjLvi32yp3sBTE2Ue2bOc0pPfoJtp2dPRRvVL12ulZV+TVIOPSA9de+ki7Of2MycmcOn7TAxmf7AdYMz01YckM77wsz01KmUelolqelzAL987DeZaTZ5fB1nBi08ODNt5tVnJPOefPJHyfTjTlk9M236NXcn8y70+z9kps289PRk3npcOyP9GVjHI6o2nFf0TcLMXgfWKXv8x7Lk7Tq9QM451wKauSXeKF7RO+eca1mt0H/g9+idc865TiBpB0kvSHpZ0nyLn0nqK+mKmP4/SUMbcV6v6J1zzrWsNjXuJ0VST+As4CvAWsDektaq2O0g4FMzWxU4Hfh9I56jV/TOOedaVieOut8IeNnMXjWzGcC/CCuflitfCfVqYDtJdU/c84reOeec63jLAW+VPR4Xt1Xdx8xmAROAxeo9sVf0zjnnWlYjW/TlK5TGn0M7+elU5aPum5ykPwBfA2YArwDfNrPxtR5v/Iy+mWk9cz4Ny8xOj0+dmAhf2iM9pZypiXn0n/RIxOIkP9TsJGV3qi2SE4I0rztu2SGfZaYNnLREMu87NjXn6Nl6rLtZMn3RtrGZabNzOgJtZnqOfuo1WWqN7LUUAPo9Vfs8+imJz0ievFCzqbnyqXnwRcy678rMtNljXkjmXcjS4Yh7bpQd8rXPuLcy0wA0YEhmWo/11kvmrUe/nCXkVu6X/Z3qCI0cdW9m5wLnZiS/DaxQ9nj5uK3aPuMk9QIGAR/XWy5v0Te/O4B1zGxd4EXguC4uj3POufZ7FBguaWVJfYC9CCuflitfCfUbwH9jjJO6eIu+ScRpFLcA9wObEa7sdjaz28t2e5jw5jvnnGuAvNHyjWJmsyR9F7iNsOjmhWb2rKQTgcfM7EbgAuDvkl4GPiFcDNTNK/rmMhzY28wOkXQlIUztZWXpBwJXdEnJnHOuG+rMlfHM7Gbg5optx5f9Pg3YvdHn9a775pIZplbSz4BZwOXVMnqYWuecc9V4Rd9cqoWpRdIBwI7Avln3azxMrXPOtZ818KdZedd9k5O0A/BjYCszm9LV5XHOue6kramr6MbwFn3z+wswELhD0hhJf+vqAnW2MX27/xfRNa96w9QuiHp9wcf8difeom8SiTC1J3RFeZrJiOmdNCzWuSrqnUe/IJr1wNVdXYRO42FqnXPOuW6sFfoLveveOeec68a8Re+cc65lede9c84514111sp4Xcm77p1zzrluzFv0zjnnWlYrzKP3ir7FLLFQdmjUN2akY8k+2jd9N2tvm56ZNsuyw+MC9E2ErlyWacm8b/Xon0wf3JbdcTUzp9uuZzqZpz5ZLDNtSs78/91nDc45esKE8enkHtnnzvvDNuuTGcn06/pnh7EdmX6r+DRRrjyDckLNJs+rdOjd1BS6VJjZInptsUf2eZdcITMNYOYtNyfT2z4el5lm09Pv46z/XpadOK32EMp58r5zU2ekw1I3Wvev5r3rfoEh6YeSTFI6QLVzzjlXxlv0CwBJKwBfAt7s6rI451x30gqj7r1F3yQkDZX0nKTzJD0r6XZJpT7p0wnr3bdCL5NzznWaNqxhP83KK/rmMhw4y8zWBsYDu0naGXjbzJ5MZfQwtc45136tEL3OK/rmUhmPfhXgp8DxeRk9TK1zzrlq/B59c6mMR78esDLwpCSA5YHRkjYys/e6oHzOOdettMI9eq/om9vTZrZb6YGk14GRZvZR1xXJOee6j2a+t94oXtG3GEt8pvPmjE/NufZtS6wl2Stn7uwkZR+7T8/0HOi8+0+pKLd5zzlv2vfUHtlnX7QDmwqzx6bHYSzatnBmWv/EmgUA0z9Mv6K9yM4/8P+2S+Zd5JgXk+kpvev4e7zOzPQ7PfPqMzLTZo95ofYTk54r33P1TZN5l591SzLdnnowM63PIb9M5p12/BGZaRqYXpuiHv1z3scZiXUvXG28om8SiXj05fsM7cQiOedct9f92/Ne0TvnnGthrXCP3vtInHPOuW7MW/TOOedalrVA571X9M4551qWd90755xzboHmLfoW8+q0gZlpCyndhbWspcNHvjQr++M0NR0Bl2mJ7jPLmQ42M31oZiSOvczs9LFzktlsifcz00Z9tFQy74Q6LrNnfzQxmT5T2dPr8lowt762XDJ97VS5l1w+mXeCap+qNjHn85nSMyfrySdnL02xkNUXMDIVajZv+tx+T56YTP/b57MXzVzmp79I5n21T3aI5cO2z/5c1yvvvejTo3Pb2K0wj95b9AsASd+T9HwMdnNKV5fHOee6i1ZY695b9E1O0jbAzsB6ZjZd0pJdXSbnnHMLDq/om4SkocAtwP3AZsDbhAr+cOBkM5sOYGYfdFUZnXOuu/Gue9fZ5gtTC6wGbCHpf5LukbRhVxbQOee6k7YG/jQrr+ibS2WY2qGEXpchwCbAj4ArFUPZlfN49M4556rxir65VIap7QWMA6614BHCheN8w4A9Hr1zzrWfNfBfs/KKvvldD2wDIGk1oA/gYWqdc64BWqHr3gfjNb8LgQslPQPMAPY3SwWbTVuCGZlpL/Xol8ybd9JlbXpm2nuWPvaiiYCx78xOh8xUXqzZhE9z8uZ9eT/4MHtdgul1lCvPQr8/O5nef4NfZablLA3AV1d7K5n+rZcXykw78LHssKkAvXPPnq2eP6Rvp5eA4LhTVs9M67nRV+s4M7R9PC4zLRVmFtLz5AEOeyJ7nn3bR+n3Uf2zP7taeHAybz2m5XwEllthfIedu1V5Rd8kcsLU7tfpBXLOuRbQzF3ujeIVvXPOuZbVzF3ujeL36J1zzrluzFv0zjnnWlZb7UOeFhhe0TvnnGtZ3b+a965755xzrlvzFr1zzrmW1Qpr3XtF3+QkjQD+BvQDZgHfiSvk1aRXItZzr5zP+6eanUxPdQ/1yDn2hB7Zxx5keeftm0xPjarNK1fetO/PZmdP0F40597f+J61zymfNeqfyfSFEqceMDtdrplT0gsALNKjT2Zaz50PTObt8+e/JtNTcqbCJ600I/2cp19zd2Zan3Hp+eh5bHr22hV9DvllMm9eTPnUXPkei6+QzDvz0pMy07TiSsm8AL2/8fPcfaoZkPOdmzwh/X1utFaYXudd983vFOBXZjYCOD4+ds455wrxFn2TSISpNWCRuNsg4J2uKJ9zznVHrTCP3iv65jIc2NvMDpF0JSFM7feB2yT9kdADs1kXls8557qVVrhH7133zaVamNrDgaPNbAXgaOCCahk9TK1zzrlqvKJvLtXC1O4PXBu3XQVsVC2jh6l1zrn28zC1rhm8A2wVf98W8Oa6c841iIepdc3gEOAMSb2AacCh9Rxs8uzst3xWzqdhw+np68LZiblobTkzyQYmwtR+1CM93WtAzjfs48S8wbac+XN5X96VFxufmfbMpCWSeUeRnTfPt3pnT3ED+Djxkn2QM63vK+umPwgHvZ4dNnjS949L5u1r6SlfKVNVe4vptT7p57zQ7/+QmaYBQ2o+L8Cs/16WmTbt+COSeV/ts1gyPRVqNjV9DqD3t7Lfq1kPX5/MW4/PcpqXU6akP9uu/byibxI5YWo36PQCOedcCzBf694555zrvnzUvXPOOecWaF7RO+eca1nNMhhP0hBJd0h6Kf6/aJV9Rkh6SNKzkp6StGeRY3tF75xzrmU10fS6Y4G7zGw4cFd8XGkK8C0zWxvYAfiTpMF5B/aK3jnnnOt6OwOXxN8vAXap3MHMXjSzl+Lv7wAfAOnpPfhgPOeccy2siQbjLWVm78bf3wOWSu0saSOgD/BK3oG9om8CknYHTgDWBDYys8fK0o4DDiKslHekmd1Wz7kG9pqZmda/LR0I9Pmc6a3Lzci+SzU9Zx79+2SH8hzWOzsN4DZlzyUG6JGYK98z5zue1+W18JDssi06IZ13NwbnHD1bry32SKZP5+majz35uezPCMBl/bKf81Y/2DeZd+KB99ZUJoBeVntY37wQzDMvPT0zrcd669V8XgCmTc1M0sDsNQkADtv+/WS6Fh6cnZYTajY1V77XJrsk89YjLzT0ssPGd9i5q2nk9DpJhzLvWifnmtm5Zel3AktXyfqzijKZlL1whKRlgL8D+5tZ7vAAr+ibwzPA14FzyjdKWgvYC1gbWBa4U9JqZjkB2p1zznW6WKmfm0jfPitN0vuSljGzd2NF/kHGfosA/wF+ZmYPFymX36PvRJKGSnpO0nlx1OTtkvqb2XNm9kKVLDsD/zKz6Wb2GvAyGWvdO+eca79mGXUP3EiIbUL8/4bKHST1Aa4DLjWzq4se2Cv6zjccOCuOmhxPCEWbZTngrbLH4+I255xzDdBEo+5PBr4o6SVg+/gYSSMlnR/32QPYEjhA0pj4MyLvwN513/mqhaKtW/m9oW8suhEewc455xYcZvYxsF2V7Y8BB8ffLwOygydk8BZ956sWijbL20B5FJDl47b5eJha55xrvzasYT/Nyiv65nYjsJekvpJWJnT7P9LFZXLOuW7DzBr206y8om8CknaVNA7YFPiPpNsAzOxZ4EpgLHArcISPuHfOOdceauarEFczf1Odc91V7QsqVLHN8l9s2N/Lu8fd0dCyNYoPxnPOOdeyGjBavul5171zzjnXjXmL3jnnXMtqa4Hb117RO+eca1ndv5r3it4551wLa+b5743i9+idc865bsxb9M4551qWt+hdp5C0e4xm1yZpZNn2L0p6XNLT8f9tu7KczjnX3bTCynjeom8OVePRAx8BXzOzdyStA9yGR69zzjnXDl7RdyJJQ4FbgPuBzQgBanY2s+di+jz7m9kTZQ+fBfpL6mtm03HOOVc377p3HaE98ejL7QaMzqrkJR0q6TFJj5177rmNKalzznVzTRSPvsN4i77ztTsevaS1gd8DX8rax8zOBUo1fPN+4pxzznUqr+g7X2U8+v6pnSUtD1wHfMvMXunIgjnnXKtp5kF0jeIVfROTNBj4D3CsmT3QxcVxzrlupxXu0XtF3wQk7QqcCSxBiEc/xsy+DHwXWBU4XtLxcfcvmdkHqeOduuJ+mWmzE0EUe+R83gfkpE9KHLtnOiuzE2l96/weTk+Uq96YkqnX8/CvfZLM+5ObBtR83mFtfZLpqZdsVs6Tnpbzh2+hxKu2aOqNBD7N+SD8+I3LMtNOWSn7c52X/7nhX03mu3bGoplp/ay+T8nMRPb+OZ/tnjnp0xLHzvu+fpYYoZX3twDgmDez36uUmR+9mkw/fYPjk+mpz4irziv6TmRmrwPrlD3+Y1nydVX2/w3wm44vmXPOtSbvunfOOee6sVbouvfpdc4551w35i1655xzLauZ5783ilf0zjnnWlZbC9yj965755xzrhvzFn2LmaTsq9el29LTiDbWxGT6zcqeLtY3Z4rSZ4n5PKvNSOftmdP19lLv7LSBdU6dWmlG9rlP/3f2lC2AvsqZi5aw3yrjkukPPL9szcd+sF/69Xxw9qeZaecunn5O132yVE1lgvxpfylPTUq/F+skWnUr9/us5vMCTJ2R/QGc0ZZua/Xp0ZZMX26F8Zlpkyf0TeadMiV7iuayw7KPW6+86XNHP35ih527mlbouvcWfRPIClNblr6ipEmSjumK8jnnXHfVZtawn2blFX1zKIWpvTcj/TRC1DvnnHOuXbzrvhO1N0xt3LYL8BowubPK6ZxzrcK77l1HKBymVtIA4CfArzqnaM4511q86951hPaEqT0BON3MJuUdtDwe/cOTXqq7kM4557oH77rvfO0JU7sx8A1JpwCDgTZJ08zsL5U7lsejP3XF/Zr30tI555pIK3Tde0XfxMxsi9Lvkk4AJlWr5J1zztWmmbvcG0WtELmnWcTBeDeZ2Trx8THAAOBJ5oapHQ+UwtSW5z2BUNGXR7wrcs5DY2u/1jLXnH9BzNuV5/bn3Hl5u/Lc/pw7L28RwxZfv2GV4Csfja438nWH8Iq+m5P0mJnNNze/M/IviHm78tz+nDsvb1ee259z5+UtYpXFP9+wSvDVj55oyoreu+6dc861LLP06oPdgY+6d84557oxb9F3f/Xe26on/4KYtyvP7c+58/J25bn9OXde3lxtLTDq3u/RO+eca1krDvlcwyrBNz95uinv0XvXvXPOOdeNede9c865ltUKXfde0TvnnGtZrXD72rvuXUNJ6ltkW7OR1F/S6l1dDleMpIW6ugy1krRkHXkXa2RZXGvwir4bkfT11E/BY4yUdJ2k0ZKekvS0pKfaUYyHCm6rdu7FJJ0Zz/24pDOK/mGTNKx0QSFpa0lHShpcMO/XgDHArfHxCEk35uRZI/6/frWfgue9TNIhpWO1l6RLyp+jpEUlXVgw71GSFlFwQXzNv1Qg39Pxc1H5U/hzIukLkhaOv+8n6TRJKxXMu5mkscDz8fF6kv6ak2eQpJMlPS/pE0kfS3oubhtc4JyLSDpJ0t8l7VORlnfuIRU/iwGPxPdqSE7ekyUtHn8fKelV4H+S3pC0VU7ekZLujp+xFSTdIWmCpEclfT7vOcdjSNLGZX9DNpaqxNLOzt+7yrbFi+Yvy/Od9uZpj1aIXudd993L1+L/SxLi3f83Pt4GeBC4tsAxLgd+BDwNFF5JQtLSwHJA//iHpPQHYRGgaOvrX8C9zA3duy9wBbB9gbzXACMlrUqYjnMD8A/gqwXyngBsBIwCMLMxklbOyfMD4FDg1CppBmxb4LwXAFsAZ0oaBjwB3GtmZxTIC7CumY2fc1KzT4v+EQcONLMzJH0ZWBT4JvB34PacfDsWPH7K2cB6ktYDfgicD1wKJCuv6HTgy8CNAGb2pKQtc/JcSfgubG1m78Gcz+v+MS3vAuci4CXCZ+xASbsB+5jZdGCTnLwfAW9UbFsOGE34nKySyPv/zOzY+PsfgD3N7FFJqxE+26nV4v4K/JIQDOtB4Ggz+6Kk7WLapqlCx4u+vxKe99tx8/LAqpK+Y2aZnxNJ2xA+S/0kjQYONbPXY/LtQOaFsKQfVG4CjpPUD8DMTkuVuxYe1MYtUMzs2wCSbgfWMrN34+NlgIsLHuZDM0u2ZjN8GTiA8Meg/Ms4EfhpwWMsY2a/Lnv8G0l7FszbZmazJO0KnGlmZ0p6omDemWY2oaKxkvz2m9mh8devmNm08rTSH6U8Zna3pHuBDQkXY4cBawNFK/oekhY1s0/jeYdQ/DtderJfBf5uZs8Waa2ZWWWlVYtZZmaSdgb+YmYXSDqoaGYze6uiqLNzsgw1s99XHOM94PeSDixwymFmVrr4vF7Sz4D/StqpQN4fAV8EfmRmTwNIes3M8i4kAXpJ6mVms4D+ZvZoLPuLyr8d1tvMbonn+72ZXR3z3iWpSLyMM4Dtyypo4rFWBm4G1kzkPQX4cvxMfQO4Q9I3zexh5n7usvwqHv/Zsn17AgMLlNll8Iq+e1qhVMlH7wMrFsz7S0nnA3dRFlLXzJK9AWZ2CXCJpN3M7Jr2Fji6XdJehFYWwDeA2wrmnSlpb0IrrdSzMV/XYYZnY5dsT0nDgSMJraAiHmT+Fkq1bfORdBewMOHWxn3Ahmb2QcHzQuhNeEjSVfHx7sBvC+Z9PF4QrkxoMQ2kQA+OpPvNbHNJE5n3YkiAmdkiBc49UdJxwH7AlpJ6UPy9ekvSZoDFruGjgOdy8rwh6cfAJWb2fnweSxEuTN8qcM6+knpYXCvVzH4r6W1C79OAVEYzO1XSFcDpkt4itLKLNiH/Ctws6WTgVklnEHrltiXcakqZFlvlgwiv1S5mdn3s8s+7MIJQN4yrsv1t8t+rPmb2LICZXS3pOeBaST8h/7mvTfhcLwz8ysymSNrfzH5VoMw1aYXBeF7Rd093SboN+Gd8vCdwZ8G83wbWIHyZS3/4jWLd/gA3xUpzKGWfLzM7sUDeQ4DvE7r9RBhDMlnS/5FfiXyb0CL+rZm9Flsefy9Y5u8BPyNc2PyTcHHx61SGilsV5ZV6e25VPAVsAKwDTADGS3rIzKYWyWxml0p6jLm3Cb5uZmMLnvsgYATwavxjuhjhNcw75+bx/3paWHsC+wAHmdl7klYkdE0XcRihtbkcodK5HTiiwPmOBe7R3IFw7xO6/3cvcM5/E17jOd8hM7tY0nuEqJNJZjYO2D32ANxBwc9H7JV6GjgcWI3wfRoOXE/O55PwOp1C+A5/GThc0kXAO4TvWZ4LgUcl/Yu5F0MrAHsRbjmlzJS0dOk2SWzZbwfcBAxLZTSzNwmv1c6EnoDTC5S1Lq0wvc5XxuumFAbfleLZ32tm1xXM94KZ1Tz6XNKthErrccpaDmZW7V72AkvS/oQW4Ujg0bKkicDFRV/veKyB8VjHAEubWaFZCpI2AZ41s4nx8SLAmmb2vwJ5RRgDsYqZnRgr26XN7JGC5/67mX0zb1szibdUdmPei1AreBFamj1Smb/QRWxZ3mHAeoSLvKJ5K8tt7cz7dULPTbues6Q1gZ0JF1UQLqxuzLuYlLQ94RbgkxXbBwHfNbNCvU4KAzZPADY2s7xxGDVbYtDqDasEP5zwQlOujOct+m4qdrUXbYWXe1DSWu1oGVZa3sx2aE8GSWuY2fPKGK1uZqMLHOM1qnQLmlnmYCdJ/66Wpyxv6h7s4oQWyk3xGKUvuBH+qOaS9F3CxdgGwOuEVtR9RfJGZzPvLYJJVbZl+SuhtbctcCLhAuUawniBItYufyCpF+F55Kro9u9D6D2aZGaDEnnOJP1eHVng1NcD4wkD4aYl96zuBuZexJZuaxWtJG4oO3ehWShlrqf2cl/P3DK3K6+ZPUf+bZFq+ar2HprZBIrfWsLMJhPGOHSoVmjsekXfjTTo/ukmwJhYcU4vy7tuwWI8KOlzpYFHBTViBHv5COR+hC7Z5PQloDQo6evA0sBl8fHehK7dlNK92dUJleMNhNfqa0ChVnEs52nA43HAVXvJyv5KmVlbrHCL2NjM1i8NWIwj9vvknjDcW/8p4ZbFZ6XNwAwKBh8p7/aPPQs7kz96/bEix87R7ovQBuZfEPNWJekWM/vKgpQ3pZmnxTWKd927OeIf3S2YfzpQ4dHWCnOcVwVqvVBoGEmPm1luK1PSY2Y2Mm9bRt57CdOgSt3nA4H/FO1qVJhiVrrFcl9ld2dO3msJUwLPjpu+A2xjZrsUyPs/whTMR2OFvwRwu5kVnWN9kpkdV7SsBY73RJFzS9rdzK7K25aR91zCjIz2XIQ2JP+Cljerd43wfb7JzJZptry1GjJweMMqwU8mvtSUXfde0bt5SHrazD5XR/6qC5+040JhM+a/B3ppgXzlfyB6EFr4h5vZegXyPkeorF+Nj1cGbjaz1BSiUt4XCPPZp8fHfYGnioxzkHQkoSejdItlV+BcM8sd4BXzLwn8mdDjYYSZEkeZ2YcF8u5LGKS2PnAJYYbDz/MqzCp/iA34yMyKjF4vHaN88abSe7WVmSXndse8o81s/bxtGXnrugitJ/+CllfSbOAeqk+H28TM+jdb3lotOmDVhlWCn056ueaKXmF67BWEv3+vA3tYnDpbZd9FgLHA9Wb23bxje9e9qzRa0oYW5+y2l5m9IWlzYLiZXRRbiskpSCWS/k4YrDSGuQP5jLCYSp7ybv9ZhD9sexQs9tHAKIWVxwSsRKiAi7iUsNJZafDdLhRfs+BgQhf6ZAjznQn3bwtV9Bam4u1VeiypP2FBm9zWrZldLulxYDvCc94l3pPNU+32ypDY7b+3mY0pcIyvlf0+i/BHbedUBklfIcz5X07Sn8uSFonHKKLebt968i9oeZ8D/s/MXqpMUJgm2Ix5a9JEo+6PBe4ys5MlHRsf/yRj318TpncW4hW9q7QxsJ+k14HJtL/V80tCC211wopivQn3vr9QIPtIwkI/tXzzDiq1yMvKUmhQnJndqjB/vrQU7fOlFnqBvL+VdAtzu9+/bWZFF+oR885pnk3+giLzHkDqSZg+tTdhYZb7SVT0khYxs89i6+ED5k7BRNIQM/skdT4z2ybjuCMJvQtFblmcb2YPVOT/QixPlncI9+l3IgwsK5lIuFDLVbRXqSPyL4B5TyB7ifTvNWneBd3OwNbx90sIt+Xmq+glbQAsRViyO/f2InhF7+b35Trz7wp8njBCGDN7J963LuIZwqC4d/N2rOJq5h9tfjUFRoIrLLzyf8ytpEZJOsfMZhY5cZwVkDszoIqLCGuXl/cG5M1RBkBh4ZN9CK3cRwgXUquY2ZScrP8gtPofp8qATdJLsmYys8ckFeq5IfRYVL5X1baVH/9J4ElJ/yj6vrjaWVxJL2tKIWE0f1PlrVUT3b5eyuYudPYeoTKfh8LiUqcSFpsqsjQ44BW9q1BP13s0w8xMksGcubBJZdPcBgJjJT3CvKvyZU5zUwgIszYwqOLe7yKEUe1FnE3oeSgFKPlm3HZwwfw1MbPTJI0CNo+bvk3+aH8kjQPeJJTxGDObqLCsal4lj5ntGP8v1NtRlMJKc8m/mJI2JQwAXELzrmm+CGGZ01TeK81sD8KtpWrTKDt9sGeLqDalsNnztksjR91LOpR5b/uda2bnlqXfSWjMVPpZ+YPyv6EVvkMYPzROxeMLeUXv5lVn1zvAlZLOAQZLOgQ4EDgvJ0+RtbezrE5ooQ5m3nu/Eym2AhiEpWfLB+39V1Lh0e/1qOwNkPQm+csVX01o/e8JzJZ0A8Xnc5fOsyvwXwtzm1GI4ra1mV2fk6/afPYhhAr8qJzT9iFcNPZi3rXLPyMMBkwpHfs55p1bLcIKcK5jdJtpgZ0hVuqZ00zNLLMVLul9ScuY2bsK8Umq3craFNhCIaLfAKCPpEk2N/hR9WM3UbeFawKSxhC73kvTnSQ91Z4Wk6QvEiKCCbjNzO4omG9hYKqF+eCrEe6Z31Kkq1bSpmbW3oVISnlHA7ub2Svx8SrA1UVGcjeapLfMbIUC+4lwP29vQvf9IMKytjeb2aQC+ceY2YiKbblT3BRWBCxnwMeEaXqF1umXtFKt944zRt236/PpilvQpgXWYuGFhjasEpw85fV6Rt3/Afi4bDDeEDP7cWL/A4CRPure1aLdXe+VzOwOhXnaveIxcgd5RfcSrlYXJaxh/iih1bpvgbxPSDqC0I0/p8vezIpEJ/sRcHfFqPvcdd87SKE/OnHA4t2EcvcGdiCMwP8rYdW+PNUGPOX+PbAQvKheU+Iftcr3KnNhJEmHE7otV9G8ce8HAg9Uz+UaYHPgANW2gFZX5W2XJlow52RCj+hBhLVM9oA5A10PM7OabyV6i97NQ9IxhMAZXwROInS9/8OKz+3+P0KoyWmEJVZLX9DcQV6l1pqk7xHCcp4i6UkrNhf+KuB5wgC1EwkXB8+ZWV53cil/X8JtAIAXio66r0VG9zeE12p/K7aCYdax+1uBoDiSLiQsq3pW3HQEoQVxQE6+p8kue6E/xApR864grO1/GCHi4IdmljWVCIV10hclfCbLuyknFryIdDVQHetidFXe9urff6WGVYJTp77hC+a45hfnct9JWdc7IS515h/hivwvAZua2Uc1nPsJQqvtdMJ0uWdVcAGfUrdzqRs3tnLvM7PMpVUl7Uf4Dvy9Yvs3gdlm9o/2PociqnR/z6NoqzlOSTuB0ANRvsBQkYuqhYFfEEbuGiGq2m/yBvRl/QEuO3eRP+KPm9kG5V3ukh41s6Lr7DvXMP36rdiwSnDatDebsqL3rntX6YuxUp9zX13SqWQv3FDpFSB39HeGo4DjgOtiJb8KoXu6iNJ9/PGS1iFMT1kysT+EebnbVdl+LeE2QodU9JUVuaSFioyYr+ICwhzyeSIFFvTVygE8knYnZ7Gd8oo8jrQvVc6PFL1Hz9z36l1J/48wRz4vLoFzHcKaZ8GcDuMVvQMaeg/0OEJgm/8x7xS5ZHQxhYVfdiqfSmdhAZwiUckAzo339n9OiDM+gNBiTeldbeCamU2OPQIdKk43u4BQ1hUV1r3/PzP7TsFDTDCzW2o8/XHMX6lX21aVpD0IMeRHEXp+zpT0o9I86By/iV3xPyTMn18E+H6xYjvXWK3Qq+0VvSv5B3AL9d8DPQf4L/A04R59IWY2O87fr9VdFtaFvpe46IvyV8brL2lhi0vQligs8JMbya0B/kRYoOhGCIvCSGpP3O2746C2a5n3oipz8R41ZilZCPN+Nyy14hXWW7iTMPUvz6dxWt8EYJuYv+j0TedcO3lF74A5saInEKZr1aO3mf0gf7eqnpB0I6FVOafyNbNrs7PMcQ3tXxnvAuBqSYeVuqQlDSUMUCu0Ql29zOytioUv2tMFv3H8v3wZzLywvnUvJRv1qOiq/5jspUsrtXtlPOc6irfonWu/W+LqUP9m3lZmkV6BfoQKo7yiMuZGd5uP6lgZz8z+KGkScK/mLt86CTjZzM5OZG2UtxSi9Vm8VXAUYUGYQixj3fmcPHOWkiV0ua9BeI1fMLMZ7TjUrZJuY+5a+XsCN6cyqI6V8ZzrKN2/mvdR967B4rzXSoWm19V4vp0Jq8TtROwCjyYC/zKzBwscYxXgQwCbG1d+ZTOr9lwaRtLiwBmEke8irB1wlJl9XDD/UsDvgGXN7CuS1iLMeMjtjZD0VcJtllfiuVcmjA8ofM8/XliVbrfcZ2bX5ey/FWGRn8OAv5UlTQT+bVUiljnn6ucVvWsacTW8swnBHdaRtC5hgN5vCuSta2W8KqutPW5muQFxupJC1LyLgJ+Z2XqSegFPFJyO+Dywo5m9HB8PA/5jZmukc87J/wPgCjN7u4Zyz1kZLw6gHG/+h8i5DuNd964hKrrN51PwPvt5hFXqzol5nopdzJkVvcJ6+qPM7CGFm90XEKJevQEckDMwrREBcWpWMRiuZALwmJndUOAQi5vZlZKOAzCzWZKK3uOfWKrko1cJLeuiBgK3S/qEsPjNVWaWDMgj6XjgSjN7Pi5QdAswApglaR8zu7Md53fOFeQVvWuUryXSkvfZyyxkZo9UDE7LGwl+FHBx/H1vYD3CqPvPE7rFt6ieDWhMQJx69CPcIy9NadsNeA1YT9I2Zvb9nPyTJS1GvM0oaRPChUIRj0m6Gbgy5t8deLR0wZN3YWZmvwJ+FXtd9gTukTTOEkE74n6/jr/vTxi8twSwGiH+tlf0znUAr+hdQ5hZI9aG/yh2IZcqrm+QH5t+ls0NerMjcGm8x32npGRUs9hqvqGebv86rQt8wcxmA0g6G7iPcN+7SDCPHxDGJQyT9ACh0syLAlfSjxASd6v4+EOgP+GCp+iFGYQIW+8RBlHmLVA0o6yL/svAP+Nzfy7ednDOdQD/crmGqBhFPR8zO63AYY4ghHhcQ9LbhNbtfjl52hRCOn5KWOXut2Vp/QucE2BXSc8CU4FbCRXw0WZ2WcH8tVqUsFhOqRW+MGG9+dmSctfaN7PRcYDb6oQBdS9YgUh/MW9dF2YKYTL3IFxcXAUcYmZjc7JNj6sWvk+YP39MWdpC9ZTHOZfNK3rXKAPzd0mLK+FtH9dh71EaAZ/jeMK88J7AjWb2LMwZ4f1qwVN/ycx+rBCj/XXg64SFdzq6oj8FGCNpFKGi3hL4XXz+ud3YkvoRVjPcnNAKv0/S38xsWoG8F1FlZpEVi/YHsALwfTMbU3B/CLdZriZcHJxemtUQZwA80Y7jOOfawUfdu6YRB2jtBgxl3iAtJ+bk6wUMjCvjlbYtTPh8F4nN/qyZrS3pfEIc+ltVMGpevWJvxEbx4aNm9k478l5JGE9QuiDZBxhsZrsXyLtb2cN+wK7AO3lLFVccY3NguJldFFfGG1BkSqKkVeJFXfm2Dp/O6Fyr8oreNVRsZR5EDXHhJd1K6MaeJ0iLmZ1a8NybMf9FwqUF8p1MmIs/lVDpDgZuMrONE9kaQtJyzB997t6Cecea2Vp52woeqwdwv5ltVnD/XxJW5FvdzFaTtCxh5H3uUrYL6nRG5xZU3nXvGu3vhLjwX6YsLnzBvMub2Q61nFTS34FhwBjmXiQYkFvRm9mxceDehHh/fDKwcy3laA+FkMB7As8yNy6AEW4bFDFa0iZm9nA83saE2xi1GE7+YLpyuxJmNowGMLN3YoyATF09ndG5VuUVvWu0Vc1sd0k7m9klcR78fQXzPijpc2ZWZMR5pZHAWrUsvBKXn90P2DJO7buHeVdu6yi7EFrEuQPvMmxAeM3eJFwgrAS8IOlpwmqE62ZllDSRee/Rv0fxUMQQR9BLKs2QWLhAnq6ezuhcS/KK3jVaLXHhSzYHDojL6E4nDFBLVlhlngGWJn86XjVnA72Bv8bH34zbDq7hWO3xajxvrRX9DoSR+6W1Au4FxudligsLrW1mb9Zy0pj/JknnAIPjokUHEhY8ytQE0xmda0le0btGqyUufMlX6jjv4sBYSY8wbzCdnbKzzLFhxcC7/0p6so6yFDWFMOr+LuYtc9EBcbsQLkauJVwU/R04z8zOTGWKLfH/ALlL5Sby706Yx/8ZoaV+vJndUfAQXTWd0bmW5BW9a7Ra4sKXHBzzPWgVMeILOKGd+5ebLWmYmb0Cc4LctCdcbK1uZN5APO11ELBJ6bWK9/wfIoR8zTNa0oZm9miN5x5NWKP+RzXk7arpjM61JK/oXaPVEhe+5FXCMrZ/jveQ7wPuLbLuu5ndoxDNbcO46RGbN156yo+AuyWVpnwNBRqx0l+SmV1S5yHEvBcks+O2IjYG9pX0BjCZ9t0mqZYfwgGK5O8d//9/hJH6EyqWPXbONZBX9K4hGjGi2swuAi6StDRh1bVjgEMpsBiPpD2APwCjCJXWmZJ+ZGZXFzj1A4RAOtsR7nHfRmgZdyhJw4GTgLWYdypi0ZC+FwH/k1QKD7sLIahPEV8uuF9H5P+3QvS8qcDhcQ5+7iI/zrna+Dx61xBqTFz48wmV3vuE1vz9wGgzywtsQ7yn/sVSKz5WHncWWfQmLjzzGXB53FR44Zl6SLof+CVwOmEU+rcJKwIe345jrM+8MeELrTAn6SCriFsv6WQzO7boueshaQhzpzMuBCxiZu91xrmdazXeoncN0aAR1YsRlrIdD3wCfFSkko96VHTVf0yIjlbEOhWLzNwtKW/d9kbob2Z3SZKF+OwnSHqcsKxvIRbC8GaG4k3YTdI0M7scQNJZdPBcdknbmtl/y3t8KrrsiwbScc61g1f0rtFqHlFtZrsCSFqT0DV8t6SeZrZ8gfPeKuk24J/x8Z7AzQXL3MiFZ9pjelyR7iVJ3wXeJsxS6Ay7ATdKaiNM0xtvZgd18Dm3Av5L9ZDG7YmY55xrB++6dw0laYyZjYgjqnckTMG6t2AX+o6EOeFbEhZVeZjQHX1hwXPvBpSWYL3PzK5L7V+W7znCFLHSvPIVgReAWbRvgFq7SNqQsGrgYEKc9kHA783sfx1xvnjOIWUPBwLXE8YoHA9gZp901LlTJO1mZtd0xbmd6+68oncNVU+AGEl/IUyzur89wV3qJWmlVHrsVu+McvQE9ip1p3fQOV4jtJ5V9n+JtWMgYKPL9aaZrdgV53auu/Oue9do9YyoPpUwcn99Sf0qI5xVI+l+M9u8ypKupelii+Qdo7Mq8hJJiwBHAMsRBi7eER//EHiKuYMCG87Miq5p0Nl8fp1zHcRb9K7h2juiOlZ85xPm2o8h/NEfQYhid5CZfdbhhe5Ekm4APiVM4duOsESwgKOsffHd6y1HTdH+Oqgs3qJ3roN4Re8arr0ViKSLCSuknWhmbXGbCEvnrmpm3ypwzr+b2TfztjUDSU+b2efi7z0J6/OvaGadNpdcGdH+2rH8bi3nfJp5e13mJAGrmVnfjjq3c63Mu+5dQ2VVIKTDxX7BzA4o3xCj0J0o6aWCp167ohy9KLYaX1coBf4h9nqM68xKPqo52l8dduzEcznnIq/oXaM1ugJJ3ruVdBzwU6C/pFIXv4AZwLkNKkOjrVdR1lLZC48raIB6ov3VpLPHQjjnAu+6dw0l6SrgSDMrXIFIugR4Bfh1+QWCpF8QunRzu98lnWRmx9VS5lYk6W7COIhaov3Ves7KAZNzkui8CxznWo5X9K6haqlA4mC8CwjBcMbEzSOAJ4CDzWx8wXMvCgxn3nXj721H8TudpPUoiydvZk910nm3qrbdzO7pjPM75zqPV/SuoeqpQCQNI6x1DzC2FDa24HkPBo4ClidcLGwCPGRm2xY9RmeTdBRwCHNXhNsVODcvnnx3IWlJ5r0oezOxu3OuRl7Ru6YRR9rvB6xsZidKWhFY2sweKZD3aUKI2ofjynxrAL8zs6/nZO0ykp4CNi2LJ78w4eKkQ1biqzj3JoS49WsCfQgxBiZ3Rve5pJ0IayYsC3wArAQ8Z2ZrJzM652pSNOiHc0kxEhuSJkr6rOxnYtnAszx/JbTE946PJwJnFcw7rTRyXVJfM3uesKxtM6snnny9/kJ4nV8C+gMHU/y1rtevCe/zi3EBn+0Iyx075zqAj7p3DWFmm8f/c2PHJ2xsZutLeiIe61NJfQrmHSdpMGHt9jskfQo0+yjveuLJ183MXo5Bg2YDF8XXvTMGNM40s48l9ZDUw8zulvSnTjivcy3JK3rXTGbGBWQM5sSUbyuSsRT5jhDq9W5CgJhbO6SUDWJmp0kaxdx48t8uGk++AabEi6gxkk4hTLPrrB6+8ZIGEOIaXC7pA2ByJ53buZbj9+hd05C0LyG87PrAJcA3gJ+b2VUF8m4CPGtmE+PjRYA1OzISXL26sswxkM/7hPvzRxMujP5qZi93wrkXJsQ/ELBvPPflZvZxR5/buVbkFb1rKnEQ3XaESuAuM3uuYL4ngPVL8/BjnPfHzGz9DitsnRbEMjvnFjw+GM81m/eB+4AHCSvGFa30VL7YTlwzv9lvTXV6mSUNl3SxpNMkLS/pFkmTJD0pacMOPncjBmw659qp2f8QuhYi6dfAAYRV8koVoAFF5sK/KulI4Oz4+DtAbpjbLtYVZb6IEHdgEeB/wPcJ8/e3IIzE37ijTtygAZvOuXbyrnvXNCS9AHzOzGbUkHdJ4M+EiwID7gK+b2YfNLaUjdMVZZY0xsxGxN9fNrNVq6V1pAUp0qBz3YG36F0zeQYYTFhEpV1i5bhXowvUkbqozOWzGCq7ywvNcGiABSnSoHMLPK/oXTM5CXhC0jMUXyf/x2Z2iqQzqRIwpSPjq9eqi8u8RlyRT8Cw+Dvx8SodeN4FNdKgcws8r+hdM7kE+D3wNMVbl2Pj/491SIk6xk+AUwhjET7t5HOv2cnnm8PMTgJO8kiDznUur+hdM5liZn9uZ549gZuAwWZ2RgeUqSO8L2lZ4NvA1nTesrdzYsJLWhl4t2zZ4P7AUh15bklrxKWJr6o2m8LMRnfk+Z1rVT4YzzUNSacRuuxvZN6u+8wKQNJYYHvgFqpUmmb2SUeUtR6SvkcYYb8K8HZ5EiEue4d2occyPAZsVhr4GFfJe8DMOmyKnaRzzezQuHJhJWvmSIPOLci8ondNo5YKIE5PO5y5lWZ5Rd8plWatJJ1tZod30bnnG2Ev6UkzW68ryuOc6zhe0btuoSsrzQWRpDuAM83sxvh4Z+BIM9uuk86/GTCUstuHZnZpZ5zbuVbjFb1rGpIGAb8Etoyb7gFONLMJiTyLmNlnkoZUS2/GrvtmIGkYcDmwHGHk/zjgW5201v3fgWHAGOaG6bVmnCHhXHfgFb1rGpKuIcylvyRu+iawnpl9PZHnJjPbUdJrhAprgem6bwYxihxmNqkTz/kcsJb5Hx/nOoVX9K5pZNw3zl2tTZKAFczszQ4sXrciaSngd8CyZvYVSWsBm5rZBZ1w7qsItwne7ehzOec8qI1rLlMllWKzI+kLwNS8TLFl+J+OLFg3dDFwG7BsfPwiYd37zrA4MFbSbZJuLP100rmdazk+j941k8OBS+K9egiLyexfMO9oSRua2aMdU7RuZ3EzuzKuVoeZzZI0Oy9Tg5zQSedxzuEVvWsuzxFWjBtGWPN+ArAL8FR2ljk2BvaV9AYwmblz0tftkJIu+CZLWoy4BK+kTQivd4czs3s64zzOucAretdMbgDGA6OZdyGZIr7c8NJ0bz8gLEw0TNIDwBLANzryhJLuN7PNJU1k3jX+Sxdli3Tk+Z1rVT4YzzUNSc+Y2Tp15N8cGG5mF0laAhhgZq81roTdS4watzqhon0B2MjMHujaUjnnGs0H47lm8qCkz9WSUdIvCcFiSsFSegOXNapg3YWknpL2lnQMsLqZPUtYuOYe4C9dWjjnXIfwFr3rcpKeJnTl9gKGA68S1rovfJ9d0hjg88BoM/t83PaU36Ofl6SLgRWARwjjGt4hxII/zsyu77qSOec6it+jd81gxwYcY4aZmaTS4LKFG3DM7mgksK6ZtUnqB7wHDDOzj7u4XM65DuIVvetypdCpdbpS0jnAYEmHAAcC5zXguN3NDDNrAzCzaZJe9Ureue7Nu+5dtyDph8AHQKmr/nYzu6MLi9SUJE0BSuvZizCV8WV8OqJz3Za36F13MYDQiv8EuIJic+9b0ZpdXQDnXOfyFr3rViStC+wJ7AaMM7Ptu7hITUvS0sBGhIGQj5rZe11cJOdcB/Dpda67+YAwwOxjYMkuLkvTknQwYeT91wkL5Tws6cCuLZVzriN4i951C5K+A+xBWOHtKuBKMxvbtaVqXpJeADYrDcSLy+E+aGard23JnHON5vfoXXexAvB9MxvT1QVZQHwMTCx7PDFuc851M96id66FSPpB/HUE8DlCfAEDdgaeMrMDuqZkzrmO4i1651rLwPj/K/Gn5IYuKItzrhN4i94555zrxrxF71wLknQ384aKBcDMtu2C4jjnOpBX9M61pmPKfu9HWHdgVheVxTnXgbzr3jkHgKRHzGyjri6Hc66xvEXvXAuSNKTsYQ9CVLtBXVQc51wH8oreudb0OHPv0c8CXgcO6rLSOOc6jFf0zrWmtYDvAJsTKvz7gMe6tETOuQ7h9+ida0GSrgQ+Ay6Pm/YBBpvZ7l1XKudcR/CK3rkWJGmsma2Vt805t+Dz6HXOtabRkjYpPZC0Md5171y35C1651qQpOeA1YE346YVgRcIA/PMzNbtqrI55xrLK3rnWpCklVLpZvZGZ5XFOdexvKJ3zjnnujG/R++cc851Y17RO+ecc92YV/TOOedcN+YVvXPOOdeNeUXvnHPOdWP/H0yRMjeMPtcWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 504x504 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 删除不需要的数据\n",
    "#for data in [data_train, data_test_a]:\n",
    "    #data.drop(['issueDate','id'], axis=1,inplace=True)\n",
    "#\"纵向用缺失值上面的值替换缺失值\"\n",
    "data_train = data_train.fillna(axis=0,method='ffill')\n",
    "x_train = data_train.drop(['isDefault','id'], axis=1)\n",
    "#计算协方差\n",
    "data_corr = x_train.corrwith(data_train.isDefault) #计算相关性\n",
    "result = pd.DataFrame(columns=['features', 'corr'])\n",
    "result['features'] = data_corr.index\n",
    "result['corr'] = data_corr.values\n",
    "# 当然也可以直接看图\n",
    "data_numeric = data_train[numerical_fea]\n",
    "correlation = data_numeric.corr()\n",
    "\n",
    "f , ax = plt.subplots(figsize = (7, 7))\n",
    "plt.title('Correlation of Numeric Features with Price',y=1,size=16)\n",
    "sns.heatmap(correlation,square = True,  vmax=0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************ 1 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.749114\tvalid_1's auc: 0.729275\n",
      "[400]\ttraining's auc: 0.764716\tvalid_1's auc: 0.730125\n",
      "[600]\ttraining's auc: 0.778489\tvalid_1's auc: 0.729928\n",
      "Early stopping, best iteration is:\n",
      "[446]\ttraining's auc: 0.768137\tvalid_1's auc: 0.730186\n",
      "[0.7301862239949224]\n",
      "************************************ 2 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.748999\tvalid_1's auc: 0.731035\n",
      "[400]\ttraining's auc: 0.764879\tvalid_1's auc: 0.731436\n",
      "[600]\ttraining's auc: 0.778506\tvalid_1's auc: 0.730823\n",
      "Early stopping, best iteration is:\n",
      "[414]\ttraining's auc: 0.765823\tvalid_1's auc: 0.731478\n",
      "[0.7301862239949224, 0.7314779648434573]\n",
      "************************************ 3 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.748145\tvalid_1's auc: 0.73253\n",
      "[400]\ttraining's auc: 0.763814\tvalid_1's auc: 0.733272\n",
      "[600]\ttraining's auc: 0.777895\tvalid_1's auc: 0.733354\n",
      "Early stopping, best iteration is:\n",
      "[475]\ttraining's auc: 0.769215\tvalid_1's auc: 0.73355\n",
      "[0.7301862239949224, 0.7314779648434573, 0.7335502065719879]\n",
      "************************************ 4 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.749417\tvalid_1's auc: 0.727507\n",
      "[400]\ttraining's auc: 0.765066\tvalid_1's auc: 0.728261\n",
      "Early stopping, best iteration is:\n",
      "[353]\ttraining's auc: 0.761647\tvalid_1's auc: 0.728349\n",
      "[0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568]\n",
      "************************************ 5 ************************************\n",
      "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
      "[LightGBM] [Warning] Unknown parameter: silent\n",
      "Training until validation scores don't improve for 200 rounds\n",
      "[200]\ttraining's auc: 0.748562\tvalid_1's auc: 0.73262\n",
      "[400]\ttraining's auc: 0.764493\tvalid_1's auc: 0.733365\n",
      "Early stopping, best iteration is:\n",
      "[394]\ttraining's auc: 0.764109\tvalid_1's auc: 0.733381\n",
      "[0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568, 0.7333810157041901]\n",
      "lgb_scotrainre_list: [0.7301862239949224, 0.7314779648434573, 0.7335502065719879, 0.7283491938614568, 0.7333810157041901]\n",
      "lgb_score_mean: 0.7313889209952029\n",
      "lgb_score_std: 0.001966415347937543\n"
     ]
    }
   ],
   "source": [
    "features = [f for f in data_train.columns if f not in ['id','issueDate','isDefault'] and '_outliers' not in f]\n",
    "x_train = data_train[features]\n",
    "x_test = data_test_a[features]\n",
    "y_train = data_train['isDefault']\n",
    "def cv_model(clf, train_x, train_y, test_x, clf_name):\n",
    "    folds = 5\n",
    "    seed = 2020\n",
    "    kf = KFold(n_splits=folds, shuffle=True, random_state=seed)\n",
    "\n",
    "    train = np.zeros(train_x.shape[0])\n",
    "    test = np.zeros(test_x.shape[0])\n",
    "\n",
    "    cv_scores = []\n",
    "\n",
    "    for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)):\n",
    "        print('************************************ {} ************************************'.format(str(i+1)))\n",
    "        trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], train_y[valid_index]\n",
    "\n",
    "        if clf_name == \"lgb\":\n",
    "            train_matrix = clf.Dataset(trn_x, label=trn_y)\n",
    "            valid_matrix = clf.Dataset(val_x, label=val_y)\n",
    "\n",
    "            params = {\n",
    "                'boosting_type': 'gbdt',\n",
    "                'objective': 'binary',\n",
    "                'metric': 'auc',\n",
    "                'min_child_weight': 5,\n",
    "                'num_leaves': 2 ** 5,\n",
    "                'lambda_l2': 10,\n",
    "                'feature_fraction': 0.8,\n",
    "                'bagging_fraction': 0.8,\n",
    "                'bagging_freq': 4,\n",
    "                'learning_rate': 0.1,\n",
    "                'seed': 2020,\n",
    "                'nthread': 28,\n",
    "                'n_jobs':24,\n",
    "                'silent': True,\n",
    "                'verbose': -1,\n",
    "            }\n",
    "\n",
    "            model = clf.train(params, train_matrix, 50000, valid_sets=[train_matrix, valid_matrix], verbose_eval=200,early_stopping_rounds=200)\n",
    "            val_pred = model.predict(val_x, num_iteration=model.best_iteration)\n",
    "            test_pred = model.predict(test_x, num_iteration=model.best_iteration)\n",
    "            \n",
    "            # print(list(sorted(zip(features, model.feature_importance(\"gain\")), key=lambda x: x[1], reverse=True))[:20])\n",
    "                \n",
    "        if clf_name == \"xgb\":\n",
    "            train_matrix = clf.DMatrix(trn_x , label=trn_y)\n",
    "            valid_matrix = clf.DMatrix(val_x , label=val_y)\n",
    "            \n",
    "            params = {'booster': 'gbtree',\n",
    "                      'objective': 'binary:logistic',\n",
    "                      'eval_metric': 'auc',\n",
    "                      'gamma': 1,\n",
    "                      'min_child_weight': 1.5,\n",
    "                      'max_depth': 5,\n",
    "                      'lambda': 10,\n",
    "                      'subsample': 0.7,\n",
    "                      'colsample_bytree': 0.7,\n",
    "                      'colsample_bylevel': 0.7,\n",
    "                      'eta': 0.04,\n",
    "                      'tree_method': 'exact',\n",
    "                      'seed': 2020,\n",
    "                      'nthread': 36,\n",
    "                      \"silent\": True,\n",
    "                      }\n",
    "            \n",
    "            watchlist = [(train_matrix, 'train'),(valid_matrix, 'eval')]\n",
    "            \n",
    "            model = clf.train(params, train_matrix, num_boost_round=50000, evals=watchlist, verbose_eval=200, early_stopping_rounds=200)\n",
    "            val_pred  = model.predict(valid_matrix, ntree_limit=model.best_ntree_limit)\n",
    "            test_pred = model.predict(test_x , ntree_limit=model.best_ntree_limit)\n",
    "                 \n",
    "        if clf_name == \"cat\":\n",
    "            params = {'learning_rate': 0.05, 'depth': 5, 'l2_leaf_reg': 10, 'bootstrap_type': 'Bernoulli',\n",
    "                      'od_type': 'Iter', 'od_wait': 50, 'random_seed': 11, 'allow_writing_files': False}\n",
    "            \n",
    "            model = clf(iterations=20000, **params)\n",
    "            model.fit(trn_x, trn_y, eval_set=(val_x, val_y),\n",
    "                      cat_features=[], use_best_model=True, verbose=500)\n",
    "            \n",
    "            val_pred  = model.predict(val_x)\n",
    "            test_pred = model.predict(test_x)\n",
    "            \n",
    "        train[valid_index] = val_pred\n",
    "        test = test_pred / kf.n_splits\n",
    "        cv_scores.append(roc_auc_score(val_y, val_pred))\n",
    "        \n",
    "        print(cv_scores)\n",
    "        \n",
    "    print(\"%s_scotrainre_list:\" % clf_name, cv_scores)\n",
    "    print(\"%s_score_mean:\" % clf_name, np.mean(cv_scores))\n",
    "    print(\"%s_score_std:\" % clf_name, np.std(cv_scores))\n",
    "    return train, test\n",
    "def lgb_model(x_train, y_train, x_test):\n",
    "    lgb_train, lgb_test = cv_model(lgb, x_train, y_train, x_test, \"lgb\")\n",
    "    return lgb_train, lgb_test\n",
    "\n",
    "def xgb_model(x_train, y_train, x_test):\n",
    "    xgb_train, xgb_test = cv_model(xgb, x_train, y_train, x_test, \"xgb\")\n",
    "    return xgb_train, xgb_test\n",
    "\n",
    "def cat_model(x_train, y_train, x_test):\n",
    "    cat_train, cat_test = cv_model(CatBoostRegressor, x_train, y_train, x_test, \"cat\")\n",
    "lgb_train, lgb_test = lgb_model(x_train, y_train, x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 报错及解决办法\n",
    "#### 1.\n",
    "ValueError: DataFrame.dtypes for data must be int, float or bool.\n",
    "Did not expect the data types in the following fields: grade, subGrade, employmentLength, earliesCreditLine\n",
    "ValueError: DataFrame.dtypes for data must be int, float or bool.\n",
    "Did not expect the data types in the following fields: grade, subGrade\n",
    "\n",
    "前面的指标转数字没有做好\n",
    "\n",
    "#### 2.\n",
    "\n",
    "[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24\n",
    "\n",
    "[LightGBM] [Warning] Unknown parameter: silent\n",
    "Training until validation scores don't improve for 200 rounds\n",
    "\n",
    "重启kernel或者jupyter lab 就解决了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
